Data processing method, device and system

Abstract

The invention discloses a data processing method, device and system, which are used for solving the problem of poor reliability of data processing in a related monitoring data processing technology. The method comprises the following steps: acquiring to-be-processed data, and recording a first timestamp when the to-be-processed data is acquired; sending the to-be-processed data and a first timestamp corresponding to the to-be-processed data as to-be-confirmed data to a check queue, and sending the to-be-processed data to a data processing service cluster, so that the data processing service cluster processes the to-be-processed data; based on the first current time, a preset timeout threshold value and the first timestamp, determining whether target to-be-confirmed data exists in the check queue or not; the to-be-confirmed data corresponding to the processed to-be-processed data is moved out of the check queue; and if the target to-be-confirmed data exists, re-processing the to-be-processed data corresponding to the target to-be-confirmed data, and moving the target to-be-confirmed data out of the check queue.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a data processing method, device and system. Background Art

[0002] Information and communications technology (ICT) architecture services such as 5th Generation Mobile Communication Technology (5G) messages and instant messaging (IM) often use monitoring systems such as Zabbix and Prometheus as system and network monitoring tools.

[0003] Taking Zabbix as an example, it has systematic monitoring data collection, preprocessing, analysis and alarm triggering, supports good scalability in the form of server-side server + proxy-side proxy, can be applied to the monitoring data collection of large-scale business groups, and is widely used in various application platforms. Among them, the monitoring data is collected by the agent software agent deployed by the monitored object, aggregated to the Zabbix server for analysis and processing into the warehouse, and monitored and visualized by the Zabbix network web component. In large-scale data scenarios, Zabbix server processes and caches data through caches such as the open source high-performance key-value storage database redis or message queue middleware to avoid excessive instantaneous connections between data processing nodes and databases, which blocks upstream and downstream service processes.

[0004] However, since the data is taken out of the queue and sent to the distributed data processing node until it is written to the database, there is a lack of persistent storage and copy protection. In the event of an abnormal termination of the processing thread, a suspended restart of the processing process, or an abnormal offline processing node, the unprocessed data is lost and cannot be recovered.

[0005] In other words, the related monitoring data processing technology has the problem of poor data processing reliability. Summary of the invention

[0006] The embodiments of the present application provide a data processing method, device and system, which can solve the problem of poor reliability of data processing in related monitoring data processing technologies.

[0007] In the first aspect, an embodiment of the present application provides a data processing method, the method comprising: obtaining data to be processed, and recording a first timestamp of the data to be processed; sending the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to an inspection queue, and sending the data to be processed to the data processing service cluster so that the data processing service cluster processes the data to be processed; through a check controller, determining whether there is target data to be confirmed in the inspection queue based on a first current time, a preset timeout threshold and the first timestamp; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; if the target data to be confirmed exists, reprocessing the data to be processed corresponding to the target data to be confirmed, and removing the target data to be confirmed from the inspection queue.

[0008] In a second aspect, an embodiment of the present application provides a data processing system, the system comprising: a data acquisition queue cluster, for acquiring data to be processed and recording a first timestamp of acquiring the data to be processed; sending the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to an inspection queue, and sending the data to be processed to a data processing service cluster; the data processing service cluster, for processing the data to be processed, and reprocessing the data to be processed corresponding to target data to be confirmed; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out of processing; an inspection queue, for storing the data to be confirmed; an inspection controller, for removing the data to be confirmed corresponding to the data to be processed that has been processed and the target data to be confirmed from the inspection queue, and, based on a first current time, a preset timeout threshold and the first timestamp, determining whether the target data to be confirmed exists in the inspection queue.

[0009] In the third aspect, an embodiment of the present application provides a data processing device, which includes: an acquisition module, used to acquire data to be processed and record a first timestamp of the data to be processed; a sending module, used to send the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue, and send the data to be processed to the data processing service cluster so that the data processing service cluster processes the data to be processed; a determination module, used to determine whether there is target data to be confirmed in the inspection queue based on a first current time, a preset timeout threshold and the first timestamp corresponding to the data to be confirmed in the inspection queue; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; a reprocessing module, used to reprocess the data to be processed corresponding to the target data to be confirmed if there is the target data to be confirmed, and remove the target data to be confirmed from the inspection queue.

[0010] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions are configured to be executed by the processor, and the executable instructions include instructions for executing the data processing method as described in the first aspect.

[0011] In a fifth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium is used to store computer-executable instructions, and the computer-executable instructions enable a computer to execute the data processing method as described in the first aspect.

[0012] In a sixth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the data processing method as described in the first aspect.

[0013] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method as described in the first aspect.

[0014] In an embodiment of the present application, the data to be processed is first obtained, and the first timestamp of the data to be processed is recorded; the data to be processed and the first timestamp corresponding to the data to be processed are sent to the inspection queue as data to be confirmed, and the data to be processed is sent to the data processing service cluster so that the data processing service cluster processes the data to be processed; then, through the inspection controller, based on the first current time, the preset timeout threshold and the first timestamp, it is determined whether there is target data to be confirmed in the inspection queue; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; then, if there is target data to be confirmed, the data to be processed corresponding to the target data to be confirmed is reprocessed, and the target data to be confirmed is removed from the inspection queue. Compared with the related monitoring data processing technologies, which lack persistent storage and copy protection, this solution adds a check queue and a check controller to implement a storage mechanism for the data to be confirmed corresponding to the data to be processed and a check mechanism for the data processing status. When an abnormal event occurs and the data to be processed is not processed, the unprocessed data can be found and reprocessed, avoiding the loss of the data to be processed and being unable to recover and process it. Therefore, the reliability of the processing of the data to be processed is improved, and the problem of poor data processing reliability in the related monitoring data processing technologies is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application; Figure 2 It is a structural diagram of a working group provided in an embodiment of the present application; Figure 3 It is a schematic diagram of classifying processing situations of data to be processed provided by an embodiment of the present application; Figure 4 It is a flowchart of another data processing method provided in an embodiment of the present application; Figure 5 is a schematic diagram of a data processing system provided by an embodiment of the present application; Figure 6 is a schematic block diagram of another data processing system provided in an embodiment of the present application; Figure 7 is a structural schematic diagram of a data processing device provided in an embodiment of the present application; Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0018] The data processing method, device and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0019] Figure 1 A data processing method provided by an embodiment of the present invention is shown, and the method can be executed by an electronic device, and the electronic device may include: a server and / or a terminal device, wherein the terminal device may be, for example, a vehicle-mounted terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps: S102: Acquire the data to be processed, and record the first timestamp of acquiring the data to be processed.

[0020] In actual applications, the business cluster can be monitored to collect monitoring data; then, the monitoring data is sent to the data collection queue cluster, and the data collection queue cluster generates the to-be-processed data corresponding to the monitoring data. Then, when receiving the data processing consumption request sent by the data processing service cluster, the to-be-processed data is taken out, and the time of taking out the to-be-processed data is recorded as the first timestamp. Specifically, an agent can be installed in the business cluster host that needs to be monitored, so as to collect the monitoring data M.

[0021] S104: Send the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue, and send the data to be processed to the data processing service cluster, so that the data processing service cluster processes the data to be processed.

[0022] The data to be confirmed can be sent to the inspection queue, and the data to be processed can be sent to the data processing service cluster.

[0023] S106: Determine, by checking the controller, whether there is target data to be confirmed in the check queue based on the first current time, the preset timeout threshold and the first timestamp.

[0024] The target to-be-confirmed data is the to-be-confirmed data corresponding to the to-be-processed data that has timed out; the to-be-confirmed data corresponding to the to-be-processed data that has been processed is removed from the inspection queue.

[0025] In actual applications, the above inspection controller and inspection queue may be in a data inspection queue cluster.

[0026] Specifically, when the difference between the first current time (hereinafter referred to as T) and the first timestamp corresponding to the data to be confirmed (hereinafter referred to as T1) is greater than or equal to a preset timeout threshold (hereinafter referred to as D), the data to be confirmed is determined to be the data to be processed with a processing timeout, and the corresponding data to be confirmed is the target data to be confirmed. The first current time is the time when the controller is checked to determine whether there is the target data to be confirmed.

[0027] In actual applications, since the pending data corresponding to the pending data that has been processed in the inspection queue has been removed, and the target pending data (the pending data corresponding to the pending data that has timed out) has also been removed, the pending data at the head of the inspection queue is always the pending data that has not been processed and is the earliest to be processed. Therefore, when judging whether the pending data corresponding to the pending data in the inspection queue has timed out, if the pending data at the head of the inspection queue has not timed out, it is generally sufficient to cyclically judge whether the pending data corresponding to the pending data at the head of the inspection queue has timed out, without judging whether the pending data corresponding to other pending data in the inspection queue has timed out.

[0028] S108: If there is target data to be confirmed, reprocess the to-be-processed data corresponding to the target data to be confirmed, and remove the target data to be confirmed from the inspection queue.

[0029] Specifically, the processing timed-out data to be processed (the processing data corresponding to the target data to be confirmed) can be re-sent to the data collection queue cluster so that the processing timed-out data to be processed can be re-processed, such as re-executing the above S102 to S108 for the monitoring data that has timed-out. The target data to be confirmed can be removed from the inspection queue by the inspection controller.

[0030] In addition, after the to-be-processed data whose processing has timed out is re-sent into the data collection queue cluster, the to-be-confirmed data corresponding to the to-be-processed data whose processing has timed out in the inspection queue can be removed.

[0031] The data processing method provided by the embodiment of the present invention first obtains the data to be processed and records the first timestamp of the data to be processed; sends the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue, and sends the data to be processed to the data processing service cluster so that the data processing service cluster processes the data to be processed; then, through the inspection controller, based on the first current time, the preset timeout threshold and the first timestamp, determines whether there is target data to be confirmed in the inspection queue; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; then, if there is target data to be confirmed, the data to be processed corresponding to the target data to be confirmed is reprocessed, and the target data to be confirmed is removed from the inspection queue. Compared with the related monitoring data processing technologies, which lack persistent storage and copy protection, this solution adds a check queue and a check controller to implement a storage mechanism for the data to be confirmed corresponding to the data to be processed and a check mechanism for the data processing status. When an abnormal event occurs and the data to be processed is not processed, the unprocessed data can be found and reprocessed, avoiding the loss of the data to be processed and being unable to recover and process it. Therefore, the reliability of the processing of the data to be processed is improved, and the problem of poor data processing reliability in the related monitoring data processing technologies is solved.

[0032] In one implementation, obtaining the data to be processed (ie, S102) may be specifically performed as follows: Steps A1 to A3: Step A1, acquiring monitoring data through a collection controller in a data collection queue cluster, and assigning a preset working group to the monitoring data.

[0033] The working group includes data processing nodes in the data processing service cluster.

[0034] Due to the distributed nature of the data processing service cluster, different nodes have different performance, health, and stability. Each unit node participating in distributed data processing in the data processing service cluster can be managed as a data processing node, and a unique collection queue node (hereinafter referred to as) data check node (hereinafter referred to as) and operation statistics node (hereinafter referred to as) can be set for each data processing node, thereby forming multiple working groups (pre-set working groups), and then processing the monitoring data in units of working groups.

[0035] Specifically, the data processing node includes a processing controller and a thread pool, and the thread pool includes multiple threads.

[0036] Step A2: Generate the data to be processed based on the monitoring data and the work group to which the monitoring data is assigned through the collection controller, and store the data to be processed into the data collection queue cluster.

[0037] Specifically, the acquisition controller generates a globally unique data identifier M based on the monitoring data M and the work group to which the monitoring data is assigned. ID , M ID It is used to identify the monitoring data and the work group to which it is assigned, and then the data identifier and the monitoring data are used as the data to be processed; afterwards, it can be used as a binary group (M ID , M) to store the data to be processed in the data collection queue cluster. Correspondingly, the data to be confirmed can be (M ID , M, T1).

[0038] For example, M ID It can be a 32-byte globally unique data identifier, wherein the first 8 bytes are used to identify the work group. In the above S106, the to-be-confirmed data corresponding to the processed to-be-processed data is removed from the check queue, which can be specifically performed as follows: according to the processed to-be-processed data (M ID , M) ID The first 8 bytes of the data are matched to the inspection queue corresponding to the working group, and then the data identifier M is matched in the inspection queue corresponding to the working group. ID The same pending confirmation data will be removed from the inspection queue.

[0039] It should be noted that the above M ID It is 32 bytes and the first 8 bytes are used to identify the working group. This is just an example and there is no specific limitation on the data identification.

[0040] Correspondingly, the reprocessing of the pending data corresponding to the target to-be-confirmed data in the above S108 can be specifically executed as follows: the monitoring data M corresponding to the pending data that has timed out of processing is re-sent to the above-mentioned acquisition controller through the inspection controller, so that the acquisition controller reallocates the working group for the monitoring data M; or the pending data that has timed out of processing can be directly sent to the original working group of the data acquisition queue cluster; and then processed by the data processing nodes in the working group.

[0041] Step A3, obtaining the to-be-processed data stored in the data collection queue cluster through the data collection queue cluster.

[0042] Accordingly, sending the data to be processed to the data processing service cluster so that the data processing service cluster processes the data to be processed (ie, S104) can be specifically performed as step A4: Step A4: sending the data to be processed to the data processing node corresponding to the working group, so that the data processing node corresponding to the working group processes the data to be processed.

[0043] In this embodiment, by pre-setting a plurality of working groups, appropriate working groups are allocated to the monitoring data, so that the reprocessing of the monitoring data is more reasonable.

[0044] In one implementation, the target module may be updated by executing the following step B1: Step B1, pushing the operation parameter information of each working group pre-stored in the operation statistics database to the target module through the operation parameter controller, so that the target module is updated accordingly based on the operation parameter information of each working group.

[0045] Among them, the operating parameter information is related to the historical processing of the data to be processed by the above working group; The target module includes one or more of the following: an acquisition controller and an inspection controller.

[0046] In this embodiment, by checking the controller, the target module of each working group is updated based on the operating parameter information of each working group (related to the historical processing status of the working group on the data to be processed), thereby realizing feedback control and improving the reliability and controllability of the monitoring data acquisition and processing process.

[0047] In one implementation, the working group further includes a data checking node, and the data checking node includes the checking queue; by checking the controller, based on the first current time, the preset timeout threshold and the first timestamp, determining whether there is target data to be confirmed in the checking queue (i.e., S106), the following step C1 may be specifically performed: Step C1, by checking the controller, based on the first current time, the preset timeout threshold corresponding to the working group and the first timestamp, it is determined whether there is target data to be confirmed in the check queue.

[0048] Optionally, the preset timeout thresholds corresponding to different working groups may be the same or different.

[0049] If the target module includes a check controller, the target module is updated accordingly based on the operating parameter information of each working group (ie, step B1), and step b1 may be specifically performed: Step b1, by checking the controller, based on the operating parameter information and the preset timeout rate of each working group, the preset timeout threshold of each working group is updated.

[0050] Specifically, the operating parameter information may include the probability of timeout processing completion and the average timeout time of timeout completion within a preset period. Based on the probability of timeout processing completion and the average timeout time of timeout completion within a preset period and the preset timeout rate , determine the new preset timeout threshold :

[0051] Among them, PO is the probability of timeout completion within the preset period, and EO is the average timeout time of timeout completion.

[0052] In this embodiment, the preset timeout threshold of each working group is updated based on the operating parameter information of each working group (related to the historical processing of the working group on the data to be processed) and the preset timeout rate by checking the controller, so as to realize feedback control. On the basis of setting up the inspection mechanism for the data processing situation, the preset timeout threshold used for the inspection mechanism can be dynamically adjusted according to the operating status of the working group, so as to realize dynamic inspection of data processing. In addition, by setting corresponding preset timeout thresholds for different working groups, it is possible to make a more reasonable judgment on whether the monitoring data has been processed for timeout, so as to make the reprocessing of the monitoring data more reasonable.

[0053] In one implementation, allocating a preset working group to the monitoring data (ie, step A1) may be specifically performed as follows: Step a1, through the collection controller, based on the preset weight of each working group, the preset total weight and the preset operation weight of each working group, a load balancing algorithm is used to determine the working group to be allocated for monitoring data.

[0054] The load balancing algorithm may be a smooth weighted round-robin method.

[0055] Specifically, in the initial stage: ; Among them, WC i is the preset running weight of each working group, W i is the preset weight of each working group, S Wi is the preset total weight.

[0056] Whenever the acquisition controller receives new monitoring data, it allocates working groups to the monitoring data through the following steps i to iii, and updates the preset operating weights of each working group: Step i: when receiving new monitoring data M, the preset operation weights of each working group are WC i新 =WC i +W i .

[0057] Step ii, traverse the preset running weight WC of each working group i新 , find WC i新 The work group g with the largest value is WC i新 WC g .

[0058] Step iii, assign the new monitoring data M to the work group g, and let WC g新 =WC g -S Wi .

[0059] If the target module includes an acquisition controller, the target module is updated accordingly based on the operating parameter information of each working group (ie, step B1), and step b2 may be specifically performed: Step b2, updating the preset weight and the preset total weight of each working group based on the operating parameter information of each working group through the acquisition controller.

[0060] Specifically, the operation parameter information of each work group includes at least: the number of data processing information with the processing result category of the first category, the number of data processing information with the processing result category of the second category, the probability of processing completion within a preset timeout threshold, and the average processing completion time within multiple preset periods. The preset period can be 1 minute, 3 minutes, 15 minutes, etc.

[0061] Based on the operating parameter information of each working group, update the preset weight and preset total weight of each working group, as follows:

[0062] in, CO3 is the number of data processing information with the processing result category of the first category (indicating that the data to be processed is processed within the preset timeout threshold) within the preset period of 3 minutes; CO4 is the number of data processing information with the processing result category of the second category (indicating that the data to be processed is processed after the timeout) within the preset period of 3 minutes; The probability of completing the process within the preset timeout threshold within a preset period of 15 minutes; The probability of completing the process within the preset timeout threshold within a preset period of 3 minutes; The average processing completion time within the preset cycle of 3 minutes.

[0063] In this embodiment, the preset weight and the preset total weight of each working group are updated based on the operating parameter information of each working group (related to the historical processing of the working group on the data to be processed) through the acquisition controller, and feedback control is realized. On the basis of using the load balancing algorithm to determine the working group to be allocated for monitoring data, the data to be processed can be dynamically allocated to different working groups according to the operating status of the working group, so as to realize dynamic load balancing. Compared with the load balancing method with relatively fixed parameters in related schemes, it can better adapt to situations such as uneven performance of data processing nodes and load fluctuations, and has greater flexibility.

[0064] In one implementation, the data collection queue cluster further includes a collection queue node; the work group further includes a collection queue node; the collection queue node includes a current limiter and a collection queue; the work group may specifically include: Figure 2 The data to be processed is stored in the data collection queue cluster (i.e., step A2), which can be specifically executed as follows: step D1: Step D1, storing the data to be processed into the collection queue corresponding to the working group; Acquiring the data to be processed and recording the first timestamp of acquiring the data to be processed (ie, S102) can be specifically performed as the following step D2: Step D2, through the flow limiter of the working group, according to the time interval determined by the flow limiter, the data to be processed is taken out from the collection queue corresponding to the working group, and the first timestamp of taking out the data to be processed from the collection queue is recorded.

[0065] The collection queue serves as a buffer before data processing, accumulating data to be processed. The data processing nodes corresponding to the work group consume data from the collection queue for processing.

[0066] In practical applications, the data processing node can actively initiate a data processing consumption request to the collection queue node. The rate limiter receives the data processing consumption request and takes out the data to be processed from the collection queue according to the time interval determined by the rate limiter.

[0067] Through the data acquisition queue cluster, the data to be processed and the first timestamp corresponding to the data to be processed are sent to the inspection queue as data to be confirmed, and the data to be processed is sent to the data processing service cluster so that the data processing service cluster processes the data to be processed (i.e., step S104), which can be specifically performed as the following step D3: Step D3, by adding a current limiter to the collection queue in the working group, a frequency control and current limiting mechanism is implemented. Compared with the mode adopted by related distributed monitoring systems in which a small amount of host resources are exclusively occupied and a large amount of host resources are reused, since the data processing node rate is usually lower than the data queue, during business peak periods, when the load is sufficient, the exclusive host will continue to run at maximum performance, affecting the life and reliability of the equipment. As the load of the reused host increases, it will have a greater impact on the service performance of other reused services. This embodiment implements active control of the data processing node load at the queue level. In the scenarios of data processing node device reuse and equipment reuse, computing resources can be reserved and continuous maximum load operation can be avoided.

[0068] In one implementation, according to the time interval determined by the current limiter, the data to be processed is taken out from the collection queue (ie, step D2), which can be specifically performed as follows: step d1: Step d1, using a current limiter, based on a preset current limiting frequency, determines a time interval, and takes out data to be processed from a collection queue according to the time interval.

[0069] Specifically, during the initialization phase, the preset current limiting frequency of the collection queue of each working group can be manually set. ; Each working group current limiter can use 1 second as the control period, and use different methods according to the different intervals of the preset current limit frequency to control the time interval between the consumption data of two adjacent collection queues: when When the query rate per second (QPS) is set, the following steps I to III are executed in a loop: Step I: record the current time T, read a piece of data to be processed Mj from the collection queue, and wait for data processing consumption request; Step II, after receiving the data processing consumption request, respond to the data processing consumption request and remove the to-be-processed data Mj from the collection queue; Step III: record the first timestamp T1 of the data to be processed taken from the collection queue and calculate the time interval , wait for d time.

[0070] when When the time interval of taking out the data to be processed is dynamically controlled, the current control cycle S=0 and the number of data processing consumption requests responded in the current control cycle CR=0 are initialized, and the following steps ① to ④ are executed cyclically: Step ①, record the current time T and check whether the current control cycle is over: when When CR=0, , otherwise, when When CR=0, ; Step ②, read a piece of data to be processed Mj from the collection queue and wait for the data processing consumption request; Step ③, after receiving the data processing consumption request, respond to the data processing consumption request, increase the CR count by one, and remove Mj from the collection queue in parallel; Step ④, record the first timestamp T1 of the data to be processed taken from the collection queue, calculate the time interval d using the following formula, and wait for d time:

[0071] when When the time interval of taking out the data to be processed during each control cycle is dynamically controlled, the current control cycle S=0 is initialized, and the following steps 1) to 3 are executed cyclically: Step 1), record the current timestamp T, update the current control cycle, when season ; Step 2), read from the collection queue at one time The system continuously responds to data processing and consumption requests. After all responses are completed, the data to be processed is removed from the collection queue. Step 3), record the first timestamp T1 of taking the data to be processed from the acquisition queue, calculate the time interval as follows, and wait for a time of d:

[0072] The target module also includes a current limiter. The target module is updated accordingly based on the operating parameter information of each working group (ie, step B1). Specifically, step b3 may be performed: Step b3: updating the preset current limiting frequency of each working group based on the operating parameter information of each working group through the current limiter.

[0073] Specifically, the operating parameter information includes the probability of loss of data to be processed within a preset period.

[0074] Based on the operating parameter information of each working group, the preset current limiting frequency of each working group is updated, and the following formula can be used:

[0075] in, It is the probability of loss of the data to be processed within the preset period.

[0076] In this embodiment, the current limiter updates the preset current limiting frequency of each working group based on the operating parameter information of each working group (related to the historical processing of the working group on the data to be processed), realizing feedback control. On the basis of adding a frequency control and current limiting mechanism to the collection queue, the preset current limiting frequency can be dynamically adjusted according to the operating status of the working group to realize dynamic control of the consumption rate. In addition, different control methods are set for different preset current limiting frequency intervals.

[0077] In one implementation, the working group further includes an operation statistics node, which includes a history queue; the history queue is used to store data record information corresponding to the data to be confirmed removed from the inspection queue corresponding to the working group by the inspected controller; the data record information includes the data to be processed, a first timestamp corresponding to the data to be processed, a second timestamp and a preset timeout threshold; the second timestamp represents the time when the data to be processed is processed; and the operation parameter information of each working group is also determined through the following steps E1 to E2: Step E1, determining the processing result category corresponding to the data record information in the history queue based on the second current time, the data record information stored in the history queue corresponding to the work group and a preset timeout tolerance value through a statistical controller.

[0078] Among them, the processing result categories include the first category, the second category and the third category; the first category represents that the data to be processed is processed within a preset timeout threshold; the second category represents that the data to be processed is processed after a timeout; the third category represents that the data to be processed is lost.

[0079] In practical applications, the second current time and the first current time may be the same or different, and the values ​​of the two depend on the time for determining the processing result category and the time for determining whether there is target data to be confirmed.

[0080] Specifically, the data record information corresponding to the to-be-confirmed data removed from the inspection queue by the inspected controller and stored in the history queue includes data record information corresponding to the to-be-processed data that has been processed and data record information corresponding to the to-be-processed data that has timed out.

[0081] Considering that there may be abnormal events such as node hardware or system failure, computing or storage resource exhaustion, high network latency, service thread termination, etc., which may cause irreversible interruption of data processing tasks, and thus fail to complete processing within the preset timeout tolerance value Dmax, that is, data loss; there may also be temporary reasons such as CPU busy, performance bottleneck, resource shortage, network interruption, etc., which may cause abnormal delay of data processing tasks and fail to be processed within the preset timeout threshold D, but complete processing within the preset timeout tolerance value Dmax, that is, timeout processing. Therefore, if Figure 3 As shown, the processing conditions (processing result categories) of the data to be processed are divided into at least three categories by the statistical controller.

[0082] Specifically, the data record information can be (M ID , M, T1, T2, D). First, according to the difference between the second timestamp (T2) and the first timestamp (T1) in the data record information, determine the processing completion delay TF = T2-T1 of each data record information, and according to whether the processing completion delay is a positive value, determine whether the to-be-processed data corresponding to the data record information has been processed; If TF<0 (not processed), the second current time T is recorded, and whether the data to be processed is lost is determined based on whether T-T1 is greater than the preset timeout tolerance value Dmax; if T-T1 is greater than the preset timeout tolerance value, the processing result category R corresponding to the data record information is determined to be the third category; if T-T1 is less than the preset timeout tolerance value, the data record information is skipped and is not processed.

[0083] If TF>0 (processing has been completed), then determine whether the data to be processed has been processed by timeout based on whether the processing completion delay TF is greater than or equal to the preset timeout threshold D; if TF is greater than or equal to the preset timeout threshold, then determine that the processing result category R corresponding to the data record information is the second category; if TF is less than the preset timeout threshold, then determine that the processing result category R corresponding to the data record information is the first category.

[0084] Step E2, storing the data record information and the processing result category as data processing information in the operation statistics database through the statistics controller, and determining the operation parameter information of each work group based on the data processing information in the operation statistics database.

[0085] Specifically, the data processing information may be (M ID , M, T1, T2, D, R), and store the data processing information in the operation statistics database. In addition, the data record information of the processing result category determined in the history queue is deleted. After that, the statistics controller can determine the operation parameter information of each work group based on the data processing information in the operation statistics database according to the preset trigger interval and the work group as a unit.

[0086] In this embodiment, by setting the historical queue to connect to the inspection queue, data record information is obtained, and then based on the set statistical controller, the operating parameter information of each work group can be determined. Furthermore, by setting the historical queue, the inspection queue mentioned above, and the collection queue, a three-level queue mechanism is formed to achieve real-time monitoring and recording of the data processing operation status of each work group.

[0087] In one implementation, before determining the processing result category corresponding to the data record information in the history queue (i.e., S104) by the statistical controller based on the second current time, the data record information stored in the history queue corresponding to the work group, and the preset timeout tolerance value, the data record information sent to the history queue can be determined by executing the following steps F1 to F2: Step F1, by checking the controller, if a message indicating that the data to be processed has been processed is received, the third current time is used as the second timestamp of the data to be processed that has been processed, and the data to be processed that has been processed, the second timestamp, the first timestamp and the preset timeout threshold are sent as data record information into the historical queue of the working group.

[0088] The third current time is the time when a message indicating that the to-be-processed data has been processed is received.

[0089] Specifically, when receiving a message indicating that the data to be processed has been processed, there are two situations: in the first situation, the data to be processed has been processed within the preset timeout threshold, and accordingly, the inspection controller can match the to-be-confirmed data corresponding to the data to be processed in the inspection queue; in the second situation, the data to be processed has been processed after the timeout, and accordingly, the inspection controller cannot match the to-be-confirmed data corresponding to the data to be processed in the inspection queue, but can match the data record information corresponding to the data to be processed in the history queue. Therefore, step E1 can be specifically performed as follows: when receiving a message indicating that the data to be processed has been processed, the inspection controller first matches the data to be processed in the inspection queue; if the match is successful, the third current time is used as the second timestamp of the processed data to be processed, and the processed data to be processed, the second timestamp, the first timestamp and the preset timeout threshold are sent as data record information to the history queue of the working group; if the match fails, the history queue is matched, and when the data record information corresponding to the data to be processed is matched in the history queue, the third current time is used as the second timestamp of the processed data to be processed to update the data record information in the history queue.

[0090] Step F2, by checking the controller, if it is determined that there is target data to be confirmed in the check queue, the preset value is used as the second timestamp of the data to be processed corresponding to the target data to be confirmed, and the data to be processed, the second timestamp, the first timestamp and the preset timeout threshold corresponding to the target data to be confirmed are sent to the historical queue of the working group as data record information.

[0091] The preset value is a value smaller than the first timestamp of the to-be-processed data corresponding to the target to-be-confirmed data.

[0092] Exemplarily, the preset value may be -1.

[0093] In one implementation, the operation parameter information includes one or more of a statistical quantity index, a statistical weight index, and a statistical average index; the operation parameter information of each working group is determined based on the data processing information by the statistical controller (i.e., step E2), which can be specifically performed as the following step e1: Step e1, determining the operating parameter information of each working group based on the data processing information through the statistical controller, and storing the operating parameter information of each working group in the operating statistical database.

[0094] Among them, the statistical quantity indicators include one or more of the total number of data processing information and the number of data processing information of different processing result categories within a preset period; the statistical proportion indicators include one or more of the probability of completing the processing within a preset timeout threshold, the probability of completing the processing within a timeout, and the probability of losing the data to be processed within the preset period; the statistical average indicators include one or more of the average processing completion time and the average timeout time for timeout completion within the preset period.

[0095] Specifically, within the preset period, the probability of completing the processing within the preset timeout threshold is PF=(CF / C)*100%; within the preset period, the probability of completing the timeout processing is PO=(CO / C)*100%; within the preset period, the probability of losing the data to be processed is PL=(CL / C)*100%; Average processing completion time within the preset period ; Average timeout time for timeout completion within the preset period ; Among them, C is the total number of data processing information within the preset period; CF is the number of data processing information with the processing result category of the first category (indicating that the data to be processed is processed within the preset timeout threshold) within the preset period; CO is the number of data processing information with the processing result category of the second category (indicating that the data to be processed is processed after the timeout) within the preset period; CL is the number of data processing information with the processing result category of the third category (indicating that the data to be processed is lost) within the preset period.

[0096] In one implementation, after the data to be processed is sent to the data processing service cluster so that the data processing service cluster processes the data to be processed (i.e., S104), the following steps G1 to G3 may be performed so that the check controller receives a message indicating that the data to be processed has been processed, thereby determining that the data to be processed has been processed: Step G1, receiving the data to be processed through the data processing service cluster, and processing the data to be processed to obtain processed data.

[0097] Step G2, writing the processed data into the monitoring database through the data processing service cluster.

[0098] Step G3, when the processed data is successfully written into the monitoring database, a message indicating that the data to be processed has been processed is sent to the inspection controller through the monitoring database, so that the inspection controller determines that the data to be processed has been processed.

[0099] Specifically, the data processing service cluster includes multiple data processing nodes. The data processing node includes a processing controller and a thread pool, and the thread pool includes multiple threads. The processing controller receives the data to be processed (MID , M), wait for the thread pool to obtain an idle thread and assign data processing tasks; the thread will process the monitoring data M to obtain the output value M', and the processed data (M ID , M') is written into the monitoring database, and the thread is automatically released after the storage is successful. After that, the monitoring database sends a message indicating that the data to be processed has been processed to the inspection controller, and the inspection controller receives the message indicating that the data to be processed has been processed, and can determine that the processing of the data to be processed has been completed.

[0100] Figure 4 FIG. 1 is a flow chart of another data processing method provided in an embodiment of the present application. Figure 4 As shown, the method includes: Step 402 , acquiring monitoring data through a collection controller in a data collection queue cluster, and assigning a preset working group to the monitoring data.

[0101] The working group includes one or more of the following: a collection queue node, a data processing node and a data inspection node and an operation statistics node in a data processing service cluster; the data inspection node includes an inspection queue, and the operation statistics node includes a history queue; the history queue is used to store data record information corresponding to the to-be-confirmed data removed from the inspection queue corresponding to the working group by the inspected controller; the data record information includes the to-be-processed data, a first timestamp corresponding to the to-be-processed data, a second timestamp, and a preset timeout threshold; the second timestamp represents the time when the to-be-processed data is processed. The data inspection queue cluster includes the inspection controller and the inspection queue.

[0102] Step 404 , through the collection controller, based on the monitoring data and the work group to which the monitoring data is assigned, generates the data to be processed, and stores the data to be processed into the collection queue node corresponding to the work group.

[0103] Step 406: fetch the data to be processed from the collection queue through the flow limiter of the working group according to the time interval determined by the flow limiter, and record the first timestamp of fetching the data to be processed from the collection queue.

[0104] Step 408, through the flow limiter, the data to be processed and the first timestamp corresponding to the data to be processed are sent as data to be confirmed to the inspection queue corresponding to the working group, and the data to be processed is sent to the data processing node corresponding to the working group so that the data processing node processes the data to be processed.

[0105] Step 410: receiving the data to be processed through the data processing node, and processing the data to be processed to obtain processed data.

[0106] The data processing node includes a processing controller and a thread pool, and the thread pool includes multiple threads. After receiving the data to be processed (including data identification and monitoring data), the processing controller waits for the thread pool to obtain an idle thread and assigns a data processing task; the thread obtains an output value after processing the monitoring data, and then obtains the processed data (including data identification and output value).

[0107] Step 412: Write the processed data into the monitoring database through the data processing node.

[0108] The thread writes the processed data (including data identifier and output value) into the monitoring database, and the thread is automatically released after the data is successfully stored in the database.

[0109] Step 414, when the processed data is successfully written into the monitoring database, a message indicating that the processing of the data to be processed is completed is sent to the inspection controller through the monitoring database, so that the inspection controller determines that the processing of the data to be processed is completed.

[0110] Step 416, through the inspection controller, if a message is received indicating that the processing of the data to be processed has been completed, the third current time is used as the second timestamp of the data to be processed that has been processed, and the processed data to be processed, the second timestamp, the first timestamp and the preset timeout threshold are sent to the historical queue of the working group as data record information; and the to-be-confirmed data corresponding to the data to be processed that has been processed is removed from the inspection queue corresponding to the working group.

[0111] Step 418, by checking the controller, based on the first current time, the preset timeout threshold corresponding to the working group and the first timestamp, it is determined whether there is target data to be confirmed in the check queue.

[0112] Step 420, if there is target data to be confirmed, then by checking the controller, the preset value is used as the second timestamp of the data to be processed corresponding to the target data to be confirmed, and the data to be processed, the second timestamp, the first timestamp and the preset timeout threshold corresponding to the target data to be confirmed are sent to the historical queue of the working group as data record information.

[0113] The preset value is a value smaller than the first timestamp of the to-be-processed data corresponding to the target to-be-confirmed data.

[0114] Step 422, the inspection controller re-sends the to-be-processed data corresponding to the target to-be-confirmed data to the data collection queue cluster so that the to-be-processed data that has timed out of processing is re-processed, and the target to-be-confirmed data in the inspection queue is deleted.

[0115] Step 424, determining the processing result category corresponding to the data record information in the history queue based on the second current time, the data record information stored in the history queue corresponding to the work group, and the preset timeout tolerance value through the statistical controller.

[0116] Among them, the processing result categories include the first category, the second category and the third category; the first category represents that the data to be processed is processed within a preset timeout threshold; the second category represents that the data to be processed is processed after a timeout; the third category represents that the data to be processed is lost.

[0117] Step 426, through the statistical controller, the data record information and the processing result category are stored as data processing information in the operation statistical database, and based on the data processing information in the operation statistical database, the operation parameter information of each working group is determined.

[0118] Step 428, the operation parameter information of each working group stored in the operation statistics database is pushed to the target module through the operation parameter controller, so that the target module is updated accordingly based on the operation parameter information of each working group.

[0119] Among them, the target module includes one or more of the following: an acquisition controller and an inspection controller; operating parameter information is related to the historical processing of the work group on the data to be processed.

[0120] It should be noted that the data processing method from step 402 to step 428 may refer to the specific implementation process of the above embodiment, which will not be described in detail here.

[0121] Corresponding to step 402 to step 428, the embodiment of the present invention further provides a schematic block diagram of a corresponding data processing system, as shown in FIG. Figure 5 As shown, it is used to execute Figure 4 Describe the data processing methods.

[0122] In this embodiment, the data to be processed is first obtained, and the first timestamp of the data to be processed is recorded; the data to be processed and the first timestamp corresponding to the data to be processed are sent to the inspection queue as data to be confirmed, and the data to be processed is sent to the data processing service cluster so that the data processing service cluster processes the data to be processed; then, through the inspection controller, based on the first current time, the preset timeout threshold and the first timestamp, it is determined whether there is target data to be confirmed in the inspection queue; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; then, if there is target data to be confirmed, the data to be processed corresponding to the target data to be confirmed is reprocessed, and the target data to be confirmed is removed from the inspection queue. Compared with the related monitoring data processing technologies, which lack persistent storage and copy protection, this solution realizes the storage mechanism of the to-be-confirmed data corresponding to the to-be-processed data and the inspection mechanism of the data processing status by adding the inspection queue and the inspection controller. When the to-be-processed data is not processed completely due to abnormal events, it can find the to-be-processed data that has not been processed completely and reprocess it, avoiding the loss of the to-be-processed data and the inability to recover and process it, thereby improving the reliability of the processing of the to-be-processed data and solving the problem of poor reliability of data processing in the related monitoring data processing technologies. In addition, a feedback closed-loop mechanism is established to monitor and count the whole process of monitoring data collection, scheduling, processing, and warehousing. Based on the real-time status of the cluster, the operating parameters of data collection load balancing, data processing consumption flow limiting, and processing and warehousing completion inspection are dynamically controlled, thereby improving the reliability and controllability of the monitoring data collection and processing process.

[0123] Corresponding to the data processing method provided in the above embodiment, based on the same technical concept, the embodiment of the present invention also provides a data processing system, such as Figure 6 As shown, the data processing system includes: a data collection queue cluster 610, a data processing service cluster 620, an inspection queue 630 and an inspection controller 640. Among them, The data collection queue cluster 610 is used to obtain the data to be processed and record the first timestamp of the data to be processed; send the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue, and send the data to be processed to the data processing service cluster; The data processing service cluster 620 is used to process the data to be processed and reprocess the data to be processed corresponding to the target data to be confirmed; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out of processing; Check queue 630, used to store data to be confirmed; The inspection controller 640 is used to remove the to-be-confirmed data corresponding to the processed to-be-processed data and the target to-be-confirmed data from the inspection queue, and determine whether there is the target to-be-confirmed data in the inspection queue based on the first current time, the preset timeout threshold and the first timestamp.

[0124] Specifically, the target data to be confirmed may be sent to the data collection queue cluster 610 by the inspection controller 640 , so that the data collection queue cluster 610 reprocesses the data to be processed corresponding to the target data to be confirmed.

[0125] In one implementation, the data collection queue cluster 610 includes a collection controller; The collection controller is used to obtain monitoring data and allocate a preset work group to the monitoring data; based on the monitoring data and the work group to which the monitoring data is allocated, generate data to be processed, and store the data to be processed in the data collection queue cluster; The working group includes data processing nodes in the data processing service cluster; The data collection queue cluster 610 is specifically used to: send the data to be processed to the data processing node corresponding to the working group; The data processing node is used to process the data to be processed.

[0126] In one implementation, the data processing system further includes an operation parameter controller.

[0127] An operation parameter controller is used to push the operation parameter information of each working group pre-stored in the operation statistics database to the target module, so that the target module is updated accordingly based on the operation parameter information of each working group; the operation parameter information is related to the historical processing of the working group on the data to be processed; Wherein, the target module includes one or more of the following: an acquisition controller and an inspection controller; In one implementation, the working group further includes a data check node, which includes a check queue; and a check controller 640, which is specifically used to: The to-be-confirmed data corresponding to the to-be-processed data that has been processed is moved out of the inspection queue corresponding to the working group, and, based on the first current time, the preset timeout threshold corresponding to the working group and the first timestamp corresponding to the to-be-confirmed data in the inspection queue, it is determined whether the to-be-processed data corresponding to the to-be-confirmed data in the inspection queue has been processed for a timeout; The inspection controller 640 is also used to: Based on the operating parameter information and preset timeout rate of each working group, the preset timeout threshold of each working group is updated.

[0128] In one implementation, the collection controller is specifically used to: determine the work group to be allocated for monitoring data using a load balancing algorithm based on the preset weight of each work group, the preset total weight and the preset operation weight of each work group; The acquisition controller is also used to update the preset weight and the preset total weight of each working group based on the operating parameter information of each working group.

[0129] In one implementation, the data collection queue cluster 610 further includes a collection queue node; the above-mentioned work group further includes a collection queue node; the collection queue node includes a current limiter and a collection queue; Collection queue, used to store data to be processed; The flow limiter is used to take out the data to be processed from the collection queue according to the time interval determined by the flow limiter, and record the first timestamp of taking out the data to be processed from the collection queue; send the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue corresponding to the working group, and send the data to be processed to the data processing node corresponding to the working group, so that the data processing node processes the data to be processed.

[0130] In one implementation, the current limiter is specifically used to: determine a time interval based on a preset current limiting frequency, and take out the data to be processed from the collection queue according to the time interval; The current limiter is further used to: update the preset current limiting frequency of each working group based on the operating parameter information of each working group.

[0131] In one implementation, the data processing system further includes an operation statistics node and a statistics controller, the operation statistics node includes a history queue; The history queue is used to store data record information corresponding to the to-be-confirmed data removed from the inspection queue corresponding to the working group by the inspected controller.

[0132] The data record information includes the data to be processed, the first timestamp corresponding to the data to be processed, the second timestamp and the preset timeout threshold; the second timestamp represents the time when the data to be processed is processed; A statistical controller is used to determine the processing result category corresponding to the data record information in the historical queue based on the second current time, the data record information stored in the historical queue corresponding to the work group and a preset timeout tolerance value; store the data record information and the processing result category as data processing information in an operation statistical database, and determine the operation parameter information of each work group based on the data processing information in the operation statistical database.

[0133] Among them, the processing result categories include the first category, the second category and the third category; the first category represents that the data to be processed is processed within a preset timeout threshold; the second category represents that the data to be processed is processed after a timeout; the third category represents that the data to be processed is lost.

[0134] In one implementation, the inspection controller 640 is also used for: if a message is received indicating that the processing of the data to be processed has been completed, then the third current time is used as the second timestamp of the data to be processed that has been processed, and the processed data to be processed, the second timestamp, the first timestamp and the preset timeout threshold are sent as data record information to the historical queue of the working group; if it is determined that there is target data to be confirmed in the inspection queue, then the preset value is used as the second timestamp of the data to be processed corresponding to the target data to be confirmed, and the data to be processed, the second timestamp, the first timestamp and the preset timeout threshold are sent as data record information to the historical queue of the working group; the preset value is a value less than the first timestamp of the data to be processed corresponding to the target data to be confirmed.

[0135] In one implementation, the operating parameter information includes one or more of a statistical quantity index, a statistical weight index, and a statistical average index; the statistical controller is specifically used to: Based on the data processing information, determine the operating parameter information of each working group, and store the operating parameter information of each working group in an operating statistics database; Among them, the statistical quantity indicators include one or more of the total number of data processing information and the number of data processing information of different processing result categories within a preset period; the statistical proportion indicators include one or more of the probability of completing the processing within a preset timeout threshold, the probability of completing the processing within a timeout, and the probability of losing the data to be processed within the preset period; the statistical average indicators include one or more of the average processing completion time and the average timeout time for timeout completion within the preset period.

[0136] In one implementation, the data processing system further includes a monitoring database.

[0137] The data processing service cluster 620 is specifically used to: receive the data to be processed, process the data to be processed, and obtain the processed data; write the processed data into the monitoring database; The monitoring database is used to store the processed data and, when the processed data is successfully written into the monitoring database, send a message to the inspection controller indicating that the processing of the data to be processed is completed, so that the inspection controller determines that the processing of the data to be processed is completed.

[0138] In this embodiment, the data to be processed is first obtained, and the first timestamp of the data to be processed is recorded; the data to be processed and the first timestamp corresponding to the data to be processed are sent to the inspection queue as data to be confirmed, and the data to be processed is sent to the data processing service cluster so that the data processing service cluster processes the data to be processed; then, through the inspection controller, based on the first current time, the preset timeout threshold and the first timestamp, it is determined whether there is target data to be confirmed in the inspection queue; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; then, if there is target data to be confirmed, the data to be processed corresponding to the target data to be confirmed is reprocessed, and the target data to be confirmed is removed from the inspection queue. Compared with the related monitoring data processing technologies, which lack persistent storage and copy protection, this solution adds a check queue and a check controller to implement a storage mechanism for the data to be confirmed corresponding to the data to be processed and a check mechanism for the data processing status. When an abnormal event occurs and the data to be processed is not processed, the unprocessed data can be found and reprocessed, avoiding the loss of the data to be processed and being unable to recover and process it. Therefore, the reliability of the processing of the data to be processed is improved, and the problem of poor data processing reliability in the related monitoring data processing technologies is solved.

[0139] Corresponding to the data processing method provided in the above embodiment, based on the same technical concept, the embodiment of the present invention also provides a data processing device, Figure 7 is a schematic diagram of the structure of a data processing device according to an embodiment of the present invention, the data processing device is used to execute Figures 1 to 4 The data processing methods described, such as Figure 7 As shown, the data processing device includes: an acquisition module 710 , a sending module 720 , a determination module 730 and a reprocessing module 740 .

[0140] The acquisition module 710 is used to acquire the data to be processed and record a first timestamp when the data to be processed is acquired; The sending module 720 is used to send the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue, and send the data to be processed to the data processing service cluster so that the data processing service cluster processes the data to be processed; The determination module 730 is used to determine whether there is target data to be confirmed in the inspection queue based on the current time, the preset timeout threshold and the first timestamp; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; The reprocessing module 740 is used to reprocess the to-be-processed data corresponding to the target to-be-confirmed data if there is target to-be-confirmed data, and remove the target to-be-confirmed data from the inspection queue.

[0141] In one implementation, the data collection queue cluster includes a collection controller; an acquisition module 710, including: An allocation unit, used to obtain monitoring data through an acquisition controller in a data acquisition queue cluster, and allocate a pre-set work group to the monitoring data; The working group includes one or more of the following: a collection queue node, a data processing node in a data processing service cluster, and a data check node; the data check node includes a check queue; A storage unit is used to generate data to be processed based on the monitoring data and the work group to which the monitoring data is assigned through the collection controller, and store the data to be processed into the data collection queue cluster; The recording unit is used to obtain the to-be-processed data stored in the data collection queue cluster through the data collection queue cluster; The sending module 720 is specifically used to send the data to be processed to the data processing node corresponding to the working group, so that the data processing node corresponding to the working group processes the data to be processed.

[0142] In one implementation, the data processing device further includes a push module.

[0143] The push module is used to: push the operation parameter information of each working group pre-stored in the operation statistics database to the target module through the operation parameter controller, so that the target module is updated accordingly based on the operation parameter information of each working group; the operation parameter information is related to the historical processing of the working group on the data to be processed; Among them, the target module includes one or more of the following: an acquisition controller and an inspection controller.

[0144] In one implementation, the working group further includes a data checking node, and the data checking node includes a checking queue; the determining module 730 is specifically used to: determine whether there is target data to be confirmed in the checking queue based on the first current time, a preset timeout threshold corresponding to the working group, and the first timestamp by checking the controller; The push module is specifically used to: update the preset timeout threshold of each working group based on the operating parameter information and preset timeout rate of each working group by checking the controller.

[0145] In one implementation, the allocation unit is specifically configured to: Through the collection controller, based on the preset weight of each working group, the preset total weight and the preset operation weight of each working group, a load balancing algorithm is used to determine the working group to be allocated for monitoring data; The push module is specifically used to update the preset weight and the preset total weight of each working group based on the operating parameter information of each working group through the acquisition controller.

[0146] In one implementation, the data collection queue cluster further includes a collection queue node; the work group further includes a collection queue node; the collection queue node includes a flow limiter and a collection queue; the recording unit includes a recording subunit. The storage unit is specifically used to: The data to be processed is stored in the collection queue corresponding to the working group; Record subunit, used to: Taking out the data to be processed from the collection queue corresponding to the working group through the flow limiter of the working group according to the time interval determined by the flow limiter, and recording the first timestamp of taking out the data to be processed from the collection queue; The sending module 720 is specifically used for: Through the flow limiter corresponding to the working group, the data to be processed and the first timestamp corresponding to the data to be processed are sent as data to be confirmed to the inspection queue corresponding to the working group, and the data to be processed is sent to the data processing node corresponding to the working group so that the data processing node processes the data to be processed.

[0147] In one implementation, the recording subunit is specifically used for: The current limiter determines the time interval based on the preset current limiting frequency, and takes out the data to be processed from the collection queue according to the time interval; The push module is specifically used to: update the preset current limiting frequency of each working group based on the operating parameter information of each working group through the current limiter.

[0148] In one implementation, the working group further includes an operation statistics node, which includes a history queue; the history queue is used to store data record information corresponding to the data to be confirmed removed from the inspection queue corresponding to the working group by the inspected controller; the data record information includes the data to be processed, the first timestamp corresponding to the data to be processed, the second timestamp and the preset timeout threshold; the second timestamp represents the time when the data to be processed is processed; the data processing device further includes: a statistics module. The statistics module includes: A category determination unit, configured to determine, through a statistical controller, a category of a processing result corresponding to the data record information in the history queue based on the second current time, the data record information stored in the history queue corresponding to the work group, and a preset timeout tolerance value; The processing result categories include the first category, the second category and the third category; the first category indicates that the data to be processed is processed within the preset timeout threshold; the second category indicates that the data to be processed is processed after the timeout; the third category indicates that the data to be processed is lost; The information determination unit is used to store the data record information and the processing result category as data processing information in the operation statistics database through the statistics controller, and determine the operation parameter information of each work group based on the data processing information in the operation statistics database.

[0149] In one implementation, the data processing device further includes an information determination module. The information determination module is used to: By checking the controller, if a message indicating that the data to be processed has been processed is received, the third current time is used as the second timestamp of the data to be processed that has been processed, and the data to be processed that has been processed, the second timestamp, the first timestamp and the preset timeout threshold are sent as data record information into the history queue of the working group; By checking the controller, if it is determined that there is target data to be confirmed in the inspection queue, the preset value is used as the second timestamp of the data to be processed corresponding to the target data to be confirmed, and the data to be processed, the second timestamp, the first timestamp and the preset timeout threshold corresponding to the target data to be confirmed are sent to the historical queue of the working group as data record information; and the data to be confirmed corresponding to the data to be processed in the inspection queue is deleted; the preset value is a value less than the first timestamp of the data to be processed corresponding to the target data to be confirmed.

[0150] In one implementation, the operation parameter information includes one or more of a statistical quantity index, a statistical weight index, and a statistical average index; the determination information unit is specifically used to: Determine the operating parameter information of each working group based on the data processing information through the statistical controller, and store the operating parameter information of each working group in the operating statistical database; Among them, the statistical quantity indicators include one or more of the total number of data processing information and the number of data processing information of different processing result categories within a preset period; the statistical proportion indicators include one or more of the probability of completing the processing within a preset timeout threshold, the probability of completing the processing within a timeout, and the probability of losing the data to be processed within the preset period; the statistical average indicators include one or more of the average processing completion time and the average timeout time for timeout completion within the preset period.

[0151] In one implementation, the data processing device further includes a message sending module. The message sending module is used to: The data to be processed is received through the data processing service cluster, and the data to be processed is processed to obtain processed data; The processed data is written into the monitoring database through the data processing service cluster; When the processed data is successfully written into the monitoring database, a message indicating that the data to be processed has been processed is sent to the inspection controller through the monitoring database, so that the inspection controller determines that the data to be processed has been processed.

[0152] In this embodiment, the data to be processed is first obtained, and the first timestamp of the data to be processed is recorded; the data to be processed and the first timestamp corresponding to the data to be processed are sent to the inspection queue as data to be confirmed, and the data to be processed is sent to the data processing service cluster so that the data processing service cluster processes the data to be processed; then, through the inspection controller, based on the first current time, the preset timeout threshold and the first timestamp, it is determined whether there is target data to be confirmed in the inspection queue; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; then, if there is target data to be confirmed, the data to be processed corresponding to the target data to be confirmed is reprocessed, and the target data to be confirmed is removed from the inspection queue. Compared with the related monitoring data processing technologies, which lack persistent storage and copy protection, this solution adds a check queue and a check controller to implement a storage mechanism for the data to be confirmed corresponding to the data to be processed and a check mechanism for the data processing status. When an abnormal event occurs and the data to be processed is not processed, the unprocessed data can be found and reprocessed, avoiding the loss of the data to be processed and being unable to recover and process it. Therefore, the reliability of the processing of the data to be processed is improved, and the problem of poor data processing reliability in the related monitoring data processing technologies is solved.

[0153] Those skilled in the art should understand that the above-mentioned data processing device can be used to implement the data processing method mentioned above, and the detailed description thereof should be similar to the description of the method part mentioned above, and will not be repeated here to avoid repetition.

[0154] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above data processing method. Figure 8A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call a computer program stored in the memory 830 and executable on the processor 810 to perform the following steps: Obtain the data to be processed, and record the first timestamp when the data to be processed is obtained; Sending the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue, and sending the data to be processed to the data processing service cluster so that the data processing service cluster processes the data to be processed; By checking the controller, the to-be-confirmed data corresponding to the to-be-processed data that has been processed is removed from the inspection queue, and, based on the first current time, the preset timeout threshold and the first timestamp, it is determined whether there is target to-be-confirmed data in the inspection queue; wherein the target to-be-confirmed data is the to-be-confirmed data corresponding to the to-be-processed data that has timed out; the to-be-confirmed data corresponding to the to-be-processed data that has been processed is removed from the inspection queue; If there is target data to be confirmed, the to-be-processed data corresponding to the target data to be confirmed is reprocessed, and the target data to be confirmed is removed from the inspection queue.

[0155] In this embodiment, the data to be processed is first obtained, and the first timestamp of the data to be processed is recorded; the data to be processed and the first timestamp corresponding to the data to be processed are sent to the inspection queue as data to be confirmed, and the data to be processed is sent to the data processing service cluster so that the data processing service cluster processes the data to be processed; then, through the inspection controller, based on the first current time, the preset timeout threshold and the first timestamp, it is determined whether there is target data to be confirmed in the inspection queue; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; then, if there is target data to be confirmed, the data to be processed corresponding to the target data to be confirmed is reprocessed, and the target data to be confirmed is removed from the inspection queue. Compared with the related monitoring data processing technologies, which lack persistent storage and copy protection, this solution adds a check queue and a check controller to implement a storage mechanism for the data to be confirmed corresponding to the data to be processed and a check mechanism for the data processing status. When an abnormal event occurs and the data to be processed is not processed, the unprocessed data can be found and reprocessed, avoiding the loss of the data to be processed and being unable to recover and process it. Therefore, the reliability of the processing of the data to be processed is improved, and the problem of poor data processing reliability in the related monitoring data processing technologies is solved.

[0156] The specific execution steps can refer to the various steps of the above-mentioned data processing method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0157] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0158] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0159] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0160] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0161] An embodiment of the present application also provides a storage medium having computer executable instructions stored thereon. When the computer executable instructions are executed by a processor, the various processes of the above-mentioned data processing method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0162] The processor is the processor in the electronic device in the above embodiment. The storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0163] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned permission management method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0164] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0165] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0166] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include multitasking and parallel processing according to the functions involved, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0167] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0168] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire the data to be processed, and record a first timestamp when the data to be processed is acquired; Sending the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to a check queue, and sending the data to be processed to a data processing service cluster so that the data processing service cluster processes the data to be processed; By checking the controller, based on the first current time, the preset timeout threshold and the first timestamp, determining whether there is target data to be confirmed in the inspection queue; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; If the target data to be confirmed exists, the to-be-processed data corresponding to the target data to be confirmed is reprocessed, and the target data to be confirmed is removed from the inspection queue.

2. The method according to claim 1, characterized in that By obtaining the data to be processed, including: Acquire monitoring data through an acquisition controller in a data acquisition queue cluster, and assign a pre-set work group to the monitoring data; wherein the work group includes a data processing node in the data processing service cluster; Generate the data to be processed based on the monitoring data and the work group to which the monitoring data is assigned, and store the data to be processed into the data collection queue cluster by the collection controller; Obtaining the to-be-processed data stored in the data acquisition queue cluster through the data acquisition queue cluster; Correspondingly, sending the data to be processed to the data processing service cluster so that the data processing service cluster processes the data to be processed includes: The data to be processed is sent to the data processing node corresponding to the working group, so that the data processing node corresponding to the working group processes the data to be processed.

3. The method according to claim 2, characterized in that The method further comprises: The operation parameter information of each working group pre-stored in the operation statistics database is pushed to the target module through the operation parameter controller, so that the target module is updated accordingly based on the operation parameter information of each working group; the operation parameter information is related to the historical processing of the data to be processed by the working group; Wherein, the target module includes one or more of the following: the acquisition controller and the inspection controller.

4. The method according to claim 3, characterized in that: The working group further includes a data check node, and the data check node includes the check queue; The checking controller determines whether there is target data to be confirmed in the checking queue based on a first current time, a preset timeout threshold and the first timestamp, including: The checking controller determines whether the target data to be confirmed exists in the checking queue based on the first current time, the preset timeout threshold corresponding to the working group, and the first timestamp; If the target module includes the inspection controller, the target module is updated accordingly based on the operating parameter information of each of the working groups, including: The preset timeout threshold of each working group is updated through the inspection controller based on the operating parameter information and the preset timeout rate of each working group.

5. The method according to claim 3, characterized in that: The step of allocating a preset working group to the monitoring data includes: By means of the acquisition controller, based on the preset weight of each working group, the preset total weight and the preset operation weight of each working group, a load balancing algorithm is adopted to determine the working group to be allocated for the monitoring data; If the target module includes the acquisition controller, the target module is updated accordingly based on the operating parameter information of each working group, including: The preset weight and the preset total weight of each working group are updated based on the operating parameter information of each working group through the acquisition controller.

6. The method according to claim 3, characterized in that: The data collection queue cluster further includes a collection queue node; the work group further includes the collection queue node; the collection queue node includes a current limiter and a collection queue; the storing the to-be-processed data into the data collection queue cluster includes: Storing the data to be processed into the collection queue corresponding to the working group; The obtaining of the data to be processed and recording a first timestamp of obtaining the data to be processed includes: Taking out the to-be-processed data from the collection queue corresponding to the working group according to the time interval determined by the flow limiter of the working group, and recording the first timestamp of taking out the to-be-processed data from the collection queue; Correspondingly, sending the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue, and sending the data to be processed to the data processing service cluster so that the data processing service cluster processes the data to be processed, includes: Through the flow limiter corresponding to the working group, the data to be processed and the first timestamp corresponding to the data to be processed are sent as the data to be confirmed to the inspection queue corresponding to the working group, and the data to be processed is sent to the data processing node corresponding to the working group, so that the data processing node processes the data to be processed.

7. The method according to claim 6, characterized in that The taking out the to-be-processed data from the collection queue according to the time interval determined by the current limiter includes: Determining the time interval by the current limiter based on a preset current limiting frequency, and taking out the to-be-processed data from the collection queue according to the time interval; The target module also includes the current limiter, and the target module is updated accordingly based on the operating parameter information of each working group, including: The preset current limiting frequency of each of the working groups is updated through the current limiter based on the operating parameter information of each of the working groups.

8. The method according to claim 2, characterized in that: The working group further includes an operation statistics node, and the operation statistics node includes a history queue; the history queue is used to store data record information corresponding to the data to be confirmed removed by the inspection controller from the inspection queue corresponding to the working group; the data record information includes the data to be processed, the first timestamp and the second timestamp corresponding to the data to be processed, and the preset timeout threshold; The second timestamp represents the time when the to-be-processed data is processed; The method further comprises: Determine, by means of a statistical controller, a processing result category corresponding to the data record information in the history queue based on a second current time, the data record information stored in the history queue corresponding to the work group, and a preset timeout tolerance value; The processing result categories include the first category, the second category and the third category; the first category indicates that the data to be processed is processed within the preset timeout threshold; the second category indicates that the data to be processed is processed after the timeout; the third category indicates that the data to be processed is lost; The data record information and the processing result category are stored as data processing information in an operation statistics database through the statistics controller, and the operation parameter information of each of the work groups is determined based on the data processing information in the operation statistics database.

9. The method according to claim 8, characterized in that Before determining the processing result category corresponding to the data record information in the history queue based on the second current time, the data record information stored in the history queue corresponding to the work group and a preset timeout tolerance value by the statistical controller, the method further includes: By checking the controller, if a message indicating that the data to be processed has been processed is received, the third current time is used as the second timestamp of the data to be processed that has been processed, and the data to be processed that has been processed, the second timestamp, the first timestamp and the preset timeout threshold are sent as the data record information into the history queue of the working group; By checking the controller, if it is determined that the target data to be confirmed exists in the inspection queue, the preset value is used as the second timestamp of the data to be processed corresponding to the target data to be confirmed, and the data to be processed, the second timestamp, the first timestamp and the preset timeout threshold corresponding to the target data to be confirmed are sent to the historical queue of the working group as the data record information; the preset value is a value smaller than the first timestamp of the data to be processed corresponding to the target data to be confirmed.

10. The method according to claim 8, characterized in that The operation parameter information includes one or more of a statistical quantity index, a statistical weight index, and a statistical average index; the operation parameter information of each working group is determined based on the data processing information in the operation statistics database, including: Determining the operating parameter information of each of the working groups based on the data processing information through the statistical controller, and storing the operating parameter information of each of the working groups in the operating statistical database; Among them, the statistical quantity indicators include one or more of the total number of data processing information and the number of data processing information of different processing result categories within a preset period; the statistical proportion indicators include one or more of the probability of processing completion within the preset timeout threshold, the probability of processing completion within the timeout, and the probability of loss of the data to be processed within the preset period; the statistical average indicators include one or more of the average processing completion time and the average timeout time of timeout completion within the preset period.

11. The method according to claim 1, characterized in that: After sending the data to be processed to the data processing service cluster so that the data processing service cluster processes the data to be processed, the method further includes: Receiving the data to be processed through the data processing service cluster, and processing the data to be processed to obtain processed data; Writing the processed data into a monitoring database through the data processing service cluster; When the processed data is successfully written into the monitoring database, a message indicating that the data to be processed has been processed is sent to the inspection controller through the monitoring database, so that the inspection controller determines that the data to be processed has been processed.

12. A data processing system, characterized in that: The system comprises: A data acquisition queue cluster is used to obtain data to be processed and record a first timestamp of the data to be processed; send the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to the inspection queue, and send the data to be processed to the data processing service cluster; The data processing service cluster is used to process the data to be processed, and to reprocess the data to be processed corresponding to the target data to be confirmed; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out of processing; A check queue, used to store the data to be confirmed; The inspection controller is used to remove the to-be-confirmed data and the target to-be-confirmed data corresponding to the processed data out of the inspection queue, and determine whether the target to-be-confirmed data exists in the inspection queue based on a first current time, a preset timeout threshold and the first timestamp.

13. The system according to claim 12, characterized in that The data acquisition queue cluster includes an acquisition controller; The acquisition controller is used to acquire monitoring data and allocate a preset working group to the monitoring data; Based on the monitoring data and the workgroup to which the monitoring data is assigned, the data to be processed is generated, and the data to be processed is stored in the data acquisition queue cluster; wherein the workgroup includes the data processing nodes in the data processing service cluster; The data acquisition queue cluster is specifically used to: send the data to be processed to the data processing node corresponding to the working group; The data processing node is used to process the data to be processed.

14. The system according to claim 13, characterized in that The system further comprises: An operation parameter controller is used to push the operation parameter information of each working group pre-stored in the operation statistics database to the target module, so that the target module is updated accordingly based on the operation parameter information of each working group; the operation parameter information is related to the historical processing of the data to be processed by the working group; Wherein, the target module includes one or more of the following: the acquisition controller and the inspection controller.

15. A data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring data to be processed and recording a first timestamp when the data to be processed is acquired; a sending module, configured to send the data to be processed and the first timestamp corresponding to the data to be processed as data to be confirmed to a check queue, and to send the data to be processed to a data processing service cluster so that the data processing service cluster processes the data to be processed; A determination module, configured to determine whether there is target data to be confirmed in the inspection queue based on a first current time, a preset timeout threshold and the first timestamp through an inspection controller; wherein the target data to be confirmed is the data to be confirmed corresponding to the data to be processed that has timed out; and the data to be confirmed corresponding to the data to be processed that has been processed is removed from the inspection queue; The reprocessing module is used for reprocessing the to-be-processed data corresponding to the target to-be-confirmed data if the target to-be-confirmed data exists, and removing the target to-be-confirmed data from the inspection queue.

16. An electronic device, characterized in that: include: processor; as well as A memory arranged to store computer executable instructions, wherein the executable instructions are configured to be executed by the processor, and the executable instructions include instructions for executing the data processing method according to any one of claims 1 to 11.

17. A storage medium, characterized in that: The storage medium is used to store computer-executable instructions, and the computer-executable instructions enable a computer to execute the data processing method according to any one of claims 1 to 11.

18. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the data processing method according to any one of claims 1 to 11.

Images