Self-adaptive point burying system and method based on deep reinforcement learning, and server

Abstract

The invention discloses a self-adaptive point burying system based on deep reinforcement learning, which comprises a Kafka module, a ClickHouse consumption cluster module and a point burying information scheduling module, and is characterized in that the Kafka module receives point burying data of a point burying interface for temporary storage through a load balancing mechanism and a distributed architecture; the ClickHouse consumption cluster module is used for carrying out cluster consumption on the burying point data temporarily stored in the Kafka module through a multi-thread and multi-node parallel processing mode; and the burying point information scheduling module is used for selecting a corresponding scheduling strategy to schedule the burying point data according to the consumption condition of the burying point data in the Kafka module and then sending the burying point data to the ClickHouse consumption cluster module. According to the system, high availability of the point burying system can be guaranteed, and real-time balanced updating of the point burying monitoring panel can be guaranteed.

Description

Technical Field

[0001] The present application belongs to the field of network technology, and in particular, relates to an adaptive tracking point system, method and server based on deep reinforcement learning. Background Art

[0002] With the rapid development of computer technology, a large number of Internet products have emerged. In addition to providing users with basic functions and services, these products are increasingly focusing on meeting user needs and enhancing product and service value, thereby bringing commercial value. Analyzing user preferences by embedding points in front-end touchpoints (such as computer web pages, APPs, etc.) is a good choice.

[0003] Traditional tracking point systems, such as the invention patent with publication number CN114817171B, are based on the preprocessing of Spark and Flink. On the basis of the built offline data warehouse and real-time data warehouse, Impala is used to establish a joint view of the offline table and the real-time table. In the process of real-time tasks, when new tracking points are generated, the above Impala joint view is updated in real time, and the joint view is updated as the data source for real-time verification after the offline table completes the parsing of T-1 data the next day. At the same time, the tracking point verification rules are defined and saved to the backend MySQL. The front end initiates a post request, the back end receives the request parsing parameters, starts the task, filters the data source, verifies according to the verification rules, and writes the verification results to MySQL for storage. When the front end monitors the corresponding result data update, it is displayed asynchronously.

[0004] The database technologies used in the above tracking systems, such as MySQL, have extremely high storage costs and cannot meet the frequent reading and writing of massive data. Although Hive supports the storage of massive data, it cannot meet the real-time query of data. In addition, in high-concurrency scenarios, the TPS (Transactions Per Second) of the upstream module of the tracking system is much higher than that of the downstream database, which may cause the service link to be paralyzed. In addition, the traditional queuing of tracking data according to priority is prone to the problem that low-priority data is difficult to be processed under massive data.

[0005] Therefore, there is an urgent need for an adaptive tracking technology solution that can solve the above problems. Summary of the invention

[0006] In order to address the deficiencies of the prior art, the present application provides an adaptive tracking system, method and server based on deep reinforcement learning, which adopts a combination of Kafka and ClickHouse to ensure the high availability of the tracking system, and adopts a deep reinforcement learning algorithm to dynamically schedule the processing order of various tracking messages according to the status of unconsumed messages in the Kafka queue.

[0007] The technical effects to be achieved by this application are achieved through the following solutions:

[0008] According to a first aspect of the present application, an adaptive tracking system based on deep reinforcement learning is provided, including a Kafka module, a ClickHouse consumer cluster module, and a tracking information scheduling module, wherein:

[0009] The Kafka module, through a load balancing mechanism and a distributed architecture, receives the tracking data of the tracking interface for temporary storage;

[0010] The ClickHouse consumption cluster module performs cluster consumption of the buried point data temporarily stored in the Kafka module through multi-threading and multi-node parallel processing;

[0011] The tracking point information scheduling module is used to select a corresponding scheduling strategy to schedule the tracking point data according to the blocking situation of the tracking point data in the Kafka module and then send it to the ClickHouse consumption cluster module.

[0012] Preferably, the corresponding scheduling strategy is selected according to the blocking situation of the buried data in the Kafka module. The specific goal is to achieve the highest overall real-time performance through the following objective function:

[0013]

[0014] Among them: F is the overall real-time performance, K i Represents the priority of the i-th buried point data, t i Represents the delay time of the i-th tracking data, and n is the number of tracking data temporarily stored in the Kafka module.

[0015] Preferably, the tracking point information scheduling module performs deep reinforcement learning training based on a Markov decision model to select a corresponding scheduling strategy according to the tracking point scheduling environment.

[0016] Preferably, when performing deep reinforcement learning training on the Markov decision model, a 5-dimensional array is used to represent the point-of-sale scheduling environment, including:

[0017] The first dimension array is the scheduling status of each tracking point data, 0 means not scheduled, and 1 means about to be scheduled;

[0018] The second dimension array is the reward value contribution of each buried point data, where positive represents the reward value and negative represents the penalty value;

[0019] The third dimension array is the delay time of each buried point data, in ms;

[0020] The fourth dimension array is the number of blocked messages for each embedding point data;

[0021] The fifth dimension array is the number of messages consumed by each embedding point data in the previous round;

[0022] After each round of scheduling, the values ​​of the 5-dimensional array are recorded to optimize the scheduling strategy.

[0023] Preferably, the reward value contribution is calculated by a reward function, and the reward function is:

[0024] R=∑T i ×K i -∑S j ×(K j +max(K))

[0025] Where: K i is the priority; T i is the priority K i The consumption ratio of this scheduling is K. i The ratio of the number of buried point data to the total number of buried point data; S j is the priority K j The hunger ratio is K j The ratio of the number of hunger buried point data to the total number of buried point data, where the hunger buried point data is the buried point data that has not been updated for more than a threshold time; max(K) is a compensation value used to amplify the weight of the hunger ratio.

[0026] Preferably, during the training process, a different starvation threshold is set for each priority level. If the delay exceeds the threshold, the corresponding buried data will enter a starvation state. The starvation threshold is calculated according to the following formula:

[0027]

[0028] Where: K i is the priority, K i ∈[1,10] and is an integer, the unit of starvation threshold is ms.

[0029] Preferably, the scheduling strategy is a consumption strategy for the ClickHouse consumption cluster module to select the buried data in sequence, and the total number of buried data selected each time does not exceed the number of ClickHouse engines in the ClickHouse consumption cluster module; the scheduling strategy is 24 situations obtained by combining high and low priorities, high and low rewards, and high and low delay times with the number of blocked messages and the number of messages consumed the previous time, respectively.

[0030] Preferably, the DQN algorithm is used for deep reinforcement learning training.

[0031] According to a second aspect of the present application, an adaptive tracking method of an adaptive tracking system based on deep reinforcement learning is provided, comprising the following steps:

[0032] Step 1: Receive the tracking data input by the tracking interface and send it to the Kafka module for temporary storage;

[0033] Step 2: According to the blocking situation of the buried data, select the corresponding scheduling strategy to schedule the temporarily stored buried data and send it to the ClickHouse consumer cluster module;

[0034] Step 3: Use the multi-threaded and multi-node parallel processing methods of the ClickHouse consumer cluster module to perform cluster consumption of the buried data.

[0035] According to a third aspect of the present application, there is provided a server, comprising: a memory and at least one processor;

[0036] The memory stores a computer program, and the at least one processor executes the computer program stored in the memory to implement the above-mentioned adaptive burying method.

[0037] According to an embodiment of the present application, the beneficial effects of the adaptive tracking system based on deep reinforcement learning of the present application are:

[0038] To solve the problem of frequent reading and writing of massive tracking data, Kafka and ClickHouse are combined. With the advantages of column storage and distributed processing, ClickHouse greatly improves the query speed of the tracking system. Kafka's load balancing mechanism and distributed architecture can also provide a good time buffer for the database in high-concurrency scenarios, ensuring the high availability of the tracking system.

[0039] To address the problem of lack of real-time response capability of tracking point monitoring in high-concurrency scenarios, a deep reinforcement learning algorithm is used to dynamically schedule the processing order of various tracking point messages based on the status of unconsumed messages in the Kafka module, avoid excessive delays in low-priority messages, and ensure real-time and balanced updates of the tracking point monitoring panel. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 This is a structural diagram of an adaptive tracking system based on deep reinforcement learning in an embodiment of the present application;

[0042] Figure 2 The change value of the five-dimensional array after the point-burying scheduling in one embodiment of the present application;

[0043] Figure 3 This is a flow chart of an adaptive tracking method based on deep reinforcement learning in an embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the structure of a server in one embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0046] like Figure 1 As shown, the adaptive tracking system based on deep reinforcement learning in one embodiment of the present application includes a Kafka module, a ClickHouse consumption cluster module, and a tracking information scheduling module, which solves the problems of high storage delay and low query efficiency of massive tracking data in high concurrency scenarios, and ensures real-time and balanced update of tracking monitoring. Among them:

[0047] The Kafka module, through load balancing mechanism and distributed architecture, can effectively withstand the instantaneous peak traffic of the upstream tracking interface, avoid the downstream database storage module from being overwhelmed, and thus ensure the high availability of the entire tracking system; temporarily store the tracking data and send it to the ClickHouse engine in the Kafka module for consumption;

[0048] The ClickHouse consumption cluster module is used to perform cluster consumption of tracking data. It consumes the tracking data temporarily stored in the Kafka module through multi-threading and multi-node parallel processing, so that the tracking data can be imported in real time and queried in near real time.

[0049] The tracking point information scheduling module is used to select the corresponding scheduling strategy according to the blocking situation of the tracking point data, schedule the tracking point data temporarily stored in the Kafka module, and then send it to the ClickHouse consumption cluster module; it realizes dynamic scheduling of the processing order of various tracking point messages to avoid excessive delays in low-priority tracking point data, and maximizes the real-time update of the tracking point system monitoring panel.

[0050] In this embodiment, when the tracking point information scheduling module performs tracking point information scheduling, it can be described as Kafka-topic (which is the basic unit of message transmission in Kafka technology, and in this embodiment, it is tracking point data) and cluster consumption by the ClickHouse consumption cluster module. During peak periods, the consumption capacity of the ClickHouse consumption cluster module is often smaller than the production capacity of the tracking point interface. Therefore, the tracking point data will be temporarily stored in the Kafka module. The role of the tracking point information scheduling module is to arrange the order in which ClickHouse consumes the tracking point data, thereby ensuring the high real-time performance of the entire tracking point system.

[0051] The following rules should be followed when scheduling tracking information:

[0052] In a consumption cluster consisting of n tracking data and m ClickHouse engines, n≥m; each ClickHouse engine can only consume one tracking data at the same time; one tracking data can be consumed by any ClickHouse engine.

[0053] In the tracking point information scheduling module, the corresponding scheduling strategy is selected according to the consumption of tracking point data in the Kafka module. The specific goal is to achieve the highest overall real-time performance through the following objective function:

[0054]

[0055] Among them: F is the overall real-time performance, K i Represents the priority of the i-th buried point data, t i Represents the delay time of the i-th tracking data, and n is the number of tracking data temporarily stored in the Kafka module.

[0056] pass The overall real-time performance of the system is calculated, that is, the sum of the real-time performance of all tracking data. The ratio of priority to delay time is used to represent the real-time performance. The higher the priority, the smaller the delay time and the higher the real-time performance. The sum of the real-time performance of all tracking data represents the overall real-time performance of the system.

[0057] The tracking point information scheduling module performs deep reinforcement learning training based on the Markov decision model, and continuously optimizes the scheduling actions until the best effect is achieved. Deep reinforcement learning is to take an action based on the current state, observe the next state and the reward obtained, and then update its own strategy based on the observed information, so as to maximize the total reward for a long time.

[0058] When performing deep reinforcement learning training on the Markov decision model, the scheduling environment, scheduling strategy, and reward function need to be modeled separately.

[0059] In terms of the tracking point scheduling environment, a 5-dimensional array is used to represent the blocking situation, including:

[0060] The first dimension array is the scheduling status of each tracking point data, 0 means not scheduled, and 1 means about to be scheduled;

[0061] The second dimension array is the reward value contribution of each buried point data, where positive represents the reward value and negative represents the penalty value;

[0062] The third dimension array is the delay time of each buried point data, in ms;

[0063] The fourth dimension array is the number of blocked messages for each embedding point data;

[0064] The fifth dimension array is the number of messages consumed by each embedding point data in the previous round;

[0065] As shown in Table 1, there is a 6×3 tracking point scheduling case, that is, there are 6 tracking point data and 3 ClickHouse engines. This case has 3 priorities. The starvation delay time of the tracking point data of the three priorities is calculated using the method of 1000 / priority, which is 1000ms, 500ms, and 333ms respectively.

[0066] Table 16×3 tracking point scheduling example

[0067]

[0068] like Figure 2 As shown in the figure, after each round of scheduling, the value of the 5-dimensional array is recorded to optimize the scheduling strategy. In the scheduling process of the 6×3 tracking scheduling case, in the initial state, all tracking data are not scheduled, so the scheduling status is 0, the reward value contribution and delay time are also 0, and the number of blocked messages is the number of message backlogs of each tracking data. In the 20th scheduling process, 3 tracking data are being scheduled. The delay time of the scheduled tracking data is 0, and the unscheduled tracking data will be delayed. The reward value contribution is calculated according to the reward function. The scheduling process will also record the number of blocked messages after each scheduling and the number of messages consumed by the tracking data this time.

[0069] The reward value contribution is calculated by the reward function, which is:

[0070] R=∑T i ×K i -∑S j ×(K j +max(K))

[0071] Where: K i is the priority; T i is the priority K iThe consumption ratio of this scheduling is K. i The ratio of the number of buried point data to the total number of buried point data; S j is the priority K j The hunger ratio is K j The ratio of the number of hungry buried point data to the total number of buried point data. The hungry buried point data refers to the buried point data that has not been updated for more than the threshold time. max(K) is the compensation value used to amplify the weight of the hungry ratio.

[0072] During the training process, a different starvation threshold is set for each priority level. If the delay exceeds the threshold, the corresponding buried data will enter a starvation state. The starvation threshold is calculated according to the following formula:

[0073]

[0074] Where: K i is the priority, K i ∈[1,10] and is an integer. The unit of starvation threshold is ms, that is, the higher the priority, the shorter the starvation time.

[0075] In this formula, T i ×K i It can well ensure that the higher the priority, the greater the proportion of buried data consumed, and the greater the reward obtained. Similarly, S i ×K i It can ensure that the lower the priority, the smaller the proportion of data starvation, and the smaller the penalty. However, simply calculating T i ×K i With S i ×K i The difference still cannot avoid the problem of starving low-priority messages to obtain the maximum R, so max(K) is used here to compensate for it. By amplifying the weight of the starvation ratio, it ensures that the reward function can evenly process low-priority data.

[0076] In traditional Kafka technology, low-priority data will be processed after high-priority data is processed. As a result, low-priority data cannot be consumed for a long time and the delay time is too long. If it is still scheduled according to priority, it will cause low-priority data starvation and extremely low reward values. The scheduling algorithm using the Markov decision model will balance the scheduling of high-priority and low-priority data to obtain the optimal solution of the reward function, thereby avoiding the occurrence of long-delay starvation.

[0077] The scheduling strategy is the consumption strategy of the ClickHouse consumer cluster module to select the buried data in turn. The total number of buried data selected each time does not exceed the number of ClickHouse engines in the ClickHouse consumer cluster module; the consumption strategy is high priority, high reward, and high delay time, which are combined with the number of blocked messages and the number of previous consumption messages in pairs to obtain 24 situations, as shown in Table 2:

[0078] Table 2 Consumption strategies

[0079] Serial number rule describe 1 KHH Select the topic with the highest priority and the most blocked messages 2 KHBL Select the topic with the highest priority and the least blocked messages; 3 KHH Select the topic with the highest priority and the most messages consumed in the previous session; 4 KHCL Select the topic with the highest priority and the least amount of messages consumed in the previous session; 5 KLBH Select the topic with the lowest priority and the most blocked messages; 6 KLBL Select the topic with the lowest priority and the least blocked messages 7 KLCH Select the topic with the lowest priority and the most messages consumed in the previous session 8 KLCL Select the topic with the lowest priority and the least previous consumption messages. 9 R Select the topic with the highest reward and the most blocked messages 10 RHBL Choose the topic with the highest reward and the least blocked messages 11 RHCH Select the topic with the highest reward and the most messages consumed in the previous session 12 RHCL Select the topic with the highest reward and the least previous consumption message 13 RLBH Select the topic with the lowest reward and the most blocked messages 14 RLBL Select the topic with the lowest reward and the least blocked messages 15 RLCH Select the topic with the lowest reward and the most messages consumed in the previous session 16 RLCL Select the topic with the lowest reward and the least previous consumption message 17 THB Select the topic with the highest latency and the most blocked messages 18 THBL Select the topic with the highest latency and the least blocked messages 19 TCH Select the topic with the highest latency and the most messages consumed in the previous session 20 THCL Select the topic with the highest latency and the least previous consumption messages. 21 TLCH Select the topic with the lowest latency and the most blocked messages 22 TLBL Select the topic with the lowest latency and the least blocked messages 23 TLCH Select the topic with the lowest latency and the most messages consumed in the previous session 24 TLCL Select the topic with the lowest latency and the least previous consumption messages.

[0080] The DQN algorithm is used for deep reinforcement learning training. The DQN algorithm training process is the process of approximating the neural network Q and the weight function. The tuple consisting of the environment state, action, and reward is used as input, and the advantage of each action in the current environment state is used as output.

[0081] The DQN algorithm introduces a deep learning network to fit the state-value function based on Q-learning. The algorithm can be divided into two parts: reinforcement learning and deep learning. The goal of the reinforcement learning part is to find an action sequence through iteration to maximize the sum of expected rewards; in the deep learning part, DQN uses a neural network to fit the state-value function to solve the problems of difficult search in various states and difficult Q-value storage.

[0082] According to the above method, the problems of uneven storage delay and low query efficiency of massive tracking data in high-concurrency scenarios are effectively solved, ensuring the real-time update of tracking monitoring.

[0083] like Figure 3 The adaptive tracking method using the adaptive tracking system based on deep reinforcement learning includes the following steps:

[0084] Step 1: Receive the tracking data input by the tracking interface and send it to the Kafka module for temporary storage;

[0085] In this step, the load balancing mechanism and distributed architecture of the Kafka module can effectively withstand the instantaneous peak traffic of the upstream tracking interface, preventing the downstream database storage module from being overwhelmed, thereby ensuring the high availability of the entire tracking system;

[0086] Step 2: According to the blocking situation of the buried data, select the corresponding scheduling strategy to schedule the temporarily stored buried data and send it to the ClickHouse consumer cluster module;

[0087] In this step, the processing order of various tracking data is dynamically scheduled according to the status of unconsumed messages in the Kafka module to avoid excessive delay of low-priority tracking data and maximize the real-time update of the tracking system monitoring panel;

[0088] Step 3: Use the multi-threaded and multi-node parallel processing methods of the ClickHouse consumption cluster module to perform cluster consumption of buried point data;

[0089] In this step, relying on excellent features such as multi-threading and multi-node parallel processing, data can be imported in real time and queried in near real time.

[0090] like Figure 4 As shown, a server in an embodiment of the present application includes: a memory 401 and at least one processor 402;

[0091] The memory 401 stores a computer program, and at least one processor 402 executes the computer program stored in the memory 401 to implement the above-mentioned adaptive burial point method.

[0092] According to an embodiment of the present application, the beneficial effects of the adaptive tracking system based on deep reinforcement learning of the present application are:

[0093] To solve the problem of frequent reading and writing of massive tracking data, Kafka and ClickHouse are combined. With the advantages of column storage and distributed processing, ClickHouse greatly improves the query speed of the tracking system, solving the problem that the database technology used in traditional tracking systems, such as MySQL, has extremely high storage costs and cannot meet the frequent reading and writing of massive data, and that Hive supports the storage of massive data but cannot meet the real-time query of data.

[0094] Kafka's load balancing mechanism and distributed architecture can also provide a good time buffer for the database in high-concurrency scenarios, ensuring the high availability of the tracking system. This solves the problem that the TPS of the upstream module of the traditional tracking system is much higher than that of the downstream database, which may cause the service link to be paralyzed.

[0095] To address the problem of lack of real-time response capability of tracking point monitoring in high-concurrency scenarios, a deep reinforcement learning algorithm is used to dynamically schedule the processing order of various tracking point messages to avoid excessive delays in low-priority messages and ensure real-time and balanced updates of the tracking point monitoring panel.

[0096] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.

[0097] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0098] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0099] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0100] For ease of description, spatially relative terms, such as "above", "above", "on the upper surface of", "above", etc., may be used herein to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0101] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0102] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The adaptive tracking system based on deep reinforcement learning is characterized by: It includes Kafka module, ClickHouse consumption cluster module and tracking point information scheduling module, among which: The Kafka module, through a load balancing mechanism and a distributed architecture, receives the tracking data of the tracking interface for temporary storage; The ClickHouse consumption cluster module performs cluster consumption of the buried point data temporarily stored in the Kafka module through multi-threading and multi-node parallel processing; The tracking point information scheduling module is used to select a corresponding scheduling strategy to schedule the tracking point data according to the blocking situation of the tracking point data in the Kafka module and then send it to the ClickHouse consumption cluster module.

2. The adaptive tracking system based on deep reinforcement learning according to claim 1 is characterized in that: Select the corresponding scheduling strategy according to the blocking situation of the buried data in the Kafka module. The specific goal is to achieve the highest overall real-time performance through the following objective function: Among them: F is the overall real-time performance, K i Represents the priority of the i-th buried point data, t i Represents the delay time of the i-th tracking data, and n is the number of tracking data temporarily stored in the Kafka module.

3. The adaptive tracking system based on deep reinforcement learning according to claim 2 is characterized in that: The tracking point information scheduling module performs deep reinforcement learning training based on the Markov decision model to select a corresponding scheduling strategy according to the tracking point scheduling environment.

4. The adaptive tracking system based on deep reinforcement learning according to claim 3 is characterized in that: When performing deep reinforcement learning training on the Markov decision model, a 5-dimensional array is used to represent the tracking point scheduling environment, including: The first dimension array is the scheduling status of each tracking point data, 0 means not scheduled, and 1 means about to be scheduled; The second dimension array is the reward value contribution of each buried point data, where positive represents the reward value and negative represents the penalty value; The third dimension array is the delay time of each buried point data, in ms; The fourth dimension array is the number of blocked messages for each embedding point data; The fifth dimension array is the number of messages consumed by each embedding point data in the previous round; After each round of scheduling, the values ​​of the 5-dimensional array are recorded to optimize the scheduling strategy.

5. The adaptive tracking system based on deep reinforcement learning according to claim 4 is characterized in that: The reward value contribution is calculated by a reward function, which is: R=∑T i ×K i -∑S j ×(K j +max(K)) Where: K i is the priority; T i is the priority K i The consumption ratio of this scheduling is K. i The ratio of the number of buried point data to the total number of buried point data; S j is the priority K j The hunger ratio is K j The ratio of the number of hunger buried point data to the total number of buried point data, where the hunger buried point data is the buried point data that has not been updated for more than a threshold time; max(K) is a compensation value used to amplify the weight of the hunger ratio.

6. The adaptive tracking system based on deep reinforcement learning according to claim 4 is characterized in that: During the training process, a different starvation threshold is set for each priority level. If the delay exceeds the threshold, the corresponding buried data will enter a starvation state. The starvation threshold is calculated according to the following formula: Where: K i is the priority, K i ∈[1,10] and is an integer, the unit of starvation threshold is ms.

7. The adaptive tracking system based on deep reinforcement learning according to claim 4 is characterized in that: The scheduling strategy is a consumption strategy for the ClickHouse consumption cluster module to select the buried data in sequence, and the total number of buried data selected each time does not exceed the number of ClickHouse engines in the ClickHouse consumption cluster module; the scheduling strategy is 24 situations obtained by combining high and low priorities, high and low rewards, and high and low delay times with the number of blocked messages and the number of messages consumed in the previous time, respectively.

8. The adaptive tracking system based on deep reinforcement learning according to claim 4 is characterized in that: The DQN algorithm is used for deep reinforcement learning training.

9. An adaptive tracking method of an adaptive tracking system based on deep reinforcement learning according to any one of claims 1 to 8, characterized in that: The steps include: Step 1: Receive the tracking data input by the tracking interface and send it to the Kafka module for temporary storage; Step 2: According to the blocking situation of the buried data, select the corresponding scheduling strategy to schedule the temporarily stored buried data and send it to the ClickHouse consumer cluster module; Step 3: Use the multi-threaded and multi-node parallel processing methods of the ClickHouse consumer cluster module to perform cluster consumption of the buried data.

10. A server, characterized in that: include: memory and at least one processor; The memory stores a computer program, and the at least one processor executes the computer program stored in the memory to implement the adaptive burying method described in claim 9.

Images