Data processing method and device for full-link tracking and computer device

Abstract

The application relates to a full-link tracking data processing method and device, computer equipment and a storage medium. The method comprises the following steps: sending full-amount data to a distributed message queue in real time; controlling a first consumer to pull a first message from the distributed message queue in real time, writing the latest time of the first message into a cache, and storing the first link identifier information of the message meeting a sampling rule in the first message; comparing the latest time of a second message with the latest time of the first message in the cache through a second consumer, reading the first link identifier information from the storage when it is determined according to the comparison result that the latest time of the second message is delayed by a predetermined time value compared with the latest time of the first message, and performing cold data processing on the second message as valid full-link tracking data when it is determined that the second message needs to be sampled according to the first link identifier information. The method can save more full-link tracking data.

Description

Technical Field

[0001] This invention relates to the field of end-to-end data processing technology, and in particular to an end-to-end tracking data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] End-to-end monitoring refers to detailed analysis and monitoring of the entire call chain between services in a software system. End-to-end monitoring systems are most commonly used in microservice architectures to monitor complex service chains, identify performance bottlenecks, and troubleshoot online faults.

[0003] Popular end-to-end monitoring solutions currently include Pinpoint, Skywalking, and Cat. Figure 1 This document provides an architectural diagram of Skywalking (a distributed tracing system). Due to the exceptionally large volume of data generated across the entire call chain, enterprise-level systems typically contain hundreds (the smallest service unit in a microservice architecture) or even tens of thousands of apps, with reported data volumes reaching terabytes or even petabytes daily. Existing solutions usually control the data volume through sampling rates, a simple and effective approach that allows configuration of the sampling rate on either the client or server side to reduce the overall data volume and decrease the overall system load.

[0004] Sampling rate can effectively control the amount of data, which is a percentage or per mille of the total data sampled. However, sampling rate has drawbacks such as having a single sampling rule, failing to retain the most valuable data, and not maximizing the value of the entire system.

[0005] Because a complete call trace is like a call tree, containing many sub-branches, with each node representing a data entry, and multiple data entries linked together by a TraceId, if a data entry contains an error, the entire call tree must be saved. Therefore, simple sampling methods cannot ensure that the most valuable data is stored, and the value of distributed end-to-end tracing services cannot be maximized. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data processing method, apparatus, computer equipment and storage medium for end-to-end tracking, which can save more end-to-end tracking data and thus store the most valuable data in the end-to-end tracking data.

[0007] To address at least one of the aforementioned technical problems, embodiments of the present invention provide a data processing method for end-to-end tracing, comprising: sending all data to a distributed message queue in real time; controlling a first consumer to pull a first message from the distributed message queue in real time, writing the latest time of the first message into a cache, and performing a sampling operation on the first message using a preset sampling rule, and storing the first link identification information of messages in the first message that meet the sampling rule; controlling a second consumer to pull a second message from the distributed message queue, comparing the latest time of the second message with the latest time of the first message in the cache, and when it is determined from the comparison result that the latest time of the second message is delayed by a predetermined time value compared with the latest time of the first message, reading the first link identification information from the storage, and when it is determined from the first link identification information that the second message needs to be sampled, processing the second message as valid end-to-end tracing data as cold data.

[0008] In one embodiment, the distributed message queue is a Kafka message queue, and / or the preset sampling rules include error operation rules and / or high-time-consuming operation rules.

[0009] In one embodiment, storing the first link identification information of messages that satisfy the sampling rules in the first message includes: writing the first link identification information of messages that satisfy the sampling rules in the first message into a Bloom filter for storage.

[0010] In one embodiment, the predetermined time value is the time value for responding to the end-to-end request.

[0011] In one embodiment, the first message is the first partition message, and the messages in the first message that satisfy the sampling rules are any one or more messages in the first partition message.

[0012] In one embodiment, a data processing method for end-to-end tracing further includes storing the first message as hot data after the sampling operation is completed.

[0013] In one embodiment, the second message is a second partition message. When it is determined that the second message needs to be sampled based on the first link identification information, the second message is treated as valid full-link tracing data and processed as cold data. This includes: obtaining the second link identification information of any message in the second partition message; when the second link identification information is the same as the first link identification information, determining that any message needs to be sampled, and treating any message as valid full-link tracing data and processing it as cold data.

[0014] A data processing device for end-to-end tracing includes: a sending module for sending all data to a distributed message queue in real time; a first processing module for controlling a first consumer to pull a first message from the distributed message queue in real time, writing the latest time of the first message into a cache, sampling the first message using a preset sampling rule, and storing the first link identification information of messages in the first message that meet the sampling rule; and a second processing module for controlling a second consumer to pull a second message from the distributed message queue, comparing the latest time of the second message with the latest time of the first message in the cache, and when the comparison result determines that the latest time of the second message is delayed by a predetermined time value compared to the latest time of the first message, reading the first link identification information from the storage, and when the first link identification information determines that the second message needs to be sampled, treating the second message as valid end-to-end tracing data for cold data processing.

[0015] In addition, embodiments of the present invention also provide a computer device, which includes: a memory, a processor, and an application program stored in the memory and executable on the processor, wherein the processor executes the application program to implement the steps of any of the methods described above.

[0016] In addition, embodiments of the present invention also provide a computer-readable storage medium having an application stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0017] In this embodiment of the invention, by implementing the above method, all data is sent to a distributed message queue in real time; the first consumer is controlled to pull the first message from the distributed message queue in real time, write the latest time of the first message into the cache, and perform sampling operation on the first message using a preset sampling rule, storing the first link identification information of the messages in the first message that meet the sampling rule; the second consumer is controlled to pull the second message from the distributed message queue, compare the latest time of the second message with the latest time of the first message in the cache, and when it is determined from the comparison result that the latest time of the second message is delayed by a predetermined time value compared with the latest time of the first message, the first link identification information is read from the storage, and when it is determined from the first link identification information that the second message needs to be sampled, the second message is treated as valid full-link tracking data for cold data processing.

[0018] Therefore, the first consumer consumes the first message in the distributed message queue in real time and records the first link identification information of messages that meet the sampling rules. When the second consumer consumes the second message in the distributed message queue, if the latest time of the second message consumed by the second consumer is delayed by a predetermined time compared to the message consumed by the first consumer (i.e., the latest time of the second message is delayed by a predetermined time value compared to the latest time of the first message), then the recorded first link identification information is used to determine whether the second message needs to be sampled. If so, the second message is treated as cold data. Thus, by having the first consumer identify messages that meet the sampling rules in real time, and then by setting a predetermined delay value, and then having the second consumer extract the messages required for end-to-end tracing that meet the sampling rules from the messages in the distributed message queue, more and more valuable end-to-end tracing data can be obtained, maximizing the value of the end-to-end system. Attached Figure Description

[0019] Figure 1 This is an application environment diagram of a data processing method for end-to-end tracking in an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram illustrating an implementation scenario of a data processing method for end-to-end tracking in an embodiment of the present invention;

[0021] Figure 3 This is a flowchart illustrating a data processing method for end-to-end tracking in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the hot and cold data separation processing of a data processing method for end-to-end tracking in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the structure of a data processing device for end-to-end tracking according to an embodiment of the present invention;

[0024] Figure 6 This is an internal structural diagram of the computer device in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This invention provides a data processing method for end-to-end tracking, applicable to, for example... Figure 1 In the system architecture shown. For example... Figure 2As shown, full data containing end-to-end tracing data is sent to a distributed message queue in real time. A first consumer and a second consumer are configured. The first consumer consumes messages from the distributed message queue in real time, while the second consumer consumes messages from the distributed message queue at a later time than the first consumer. The first consumer pulls the first message from the distributed message queue for consumption and stores the message's time 't' in a cache. Simultaneously, it samples the first message using a preset sampling rule, storing the first-link identifier information of messages in the first message that meet the sampling rule in storage area 1. The second consumer pulls the second message from the distributed message queue and compares the latest time of the second message with the latest time of the first message in the cache. When the comparison determines that the latest time of the second message is delayed by a predetermined time value compared to the latest time of the first message, it reads the first-link identifier information from storage area 1. If the first-link identifier information indicates that the second message requires sampling, it treats the second message as cold data, confirming it as valid end-to-end tracing data and storing it as cold data in storage area 2. Storage area 1 and storage area 2 can be the same storage area.

[0027] This application provides a data processing method for end-to-end tracing, applicable to, for example... Figure 2 In the system architecture shown. For example... Figure 3 As shown, this end-to-end tracking data processing method includes the following steps:

[0028] S302 sends all data to a distributed message queue in real time.

[0029] In this embodiment, the full data includes one or more pieces of data from the entire call chain. The distributed message queue can be a Kafka message queue. Kafka is a high-throughput distributed publish-subscribe messaging system. Consumers of any distributed message queue can subscribe to messages from the distributed message queue.

[0030] In the end-to-end call service, all generated data is sent to a distributed message queue in real time. Specifically, all end-to-end tracing data is temporarily stored in the distributed message queue after being reported. Therefore, the distributed message queue can contain real-time data generated during the end-to-end call process.

[0031] S304 controls the first consumer to pull the first message from the distributed message queue in real time, write the latest time of the first message to the cache, sample the first message using preset sampling rules, and store the first link identification information of messages in the first message that meet the sampling rules.

[0032] In this embodiment, a first consumer is set up for the distributed message queue. The first consumer is used to pull the first message from the distributed message queue in real time. This can be achieved by the first consumer subscribing to messages in the distributed message queue. When the first message is received in the distributed message queue, the first consumer pulls it from the queue through the subscription method. Therefore, the first consumer can obtain the first message generated by the end-to-end call in a timely manner.

[0033] The first consumer retrieves the latest time of the first message from its timestamp and writes the time to its cache. Furthermore, it samples the first message using a preset sampling rule. If a message in the first message matches the sampling rule, it records the first link identifier information of that message. The first link identifier information can be the message's traceId.

[0034] In one embodiment, the preset sampling rules include error operation rules and / or high-time-consuming operation rules.

[0035] Specifically, the sampling rules include error operation rules. Sampling the first message using error operation rules can identify erroneous link data in the full dataset; that is, the first link identification information of messages in the first message that meet the error operation rules is stored. The sampling rules also include high-time-consuming operation rules. Sampling the first message using high-time-consuming operation rules can identify link data with significant time consumption in the full dataset; that is, the first link identification information of messages in the first message that meet the high-time-consuming operation rules is stored.

[0036] Therefore, after sampling, valuable data can be identified, such as faulty link data, time-consuming link data, core link data, and other data that meet the sampling rules. This data is of great value for observing performance bottlenecks and troubleshooting.

[0037] In one embodiment, storing the first link identifier information of the message that satisfies the sampling rule in the first message includes: writing the first link identifier information of the message that satisfies the sampling rule in the first message into a Bloom filter for storage.

[0038] Specifically, the message that matches the sampling rule is trace data, and the first link identification information is the traceId of the trace data. The traceId of the trace data that matches the sampling rule is written into the Redis Bloom filter.

[0039] In one embodiment, the first message is the first partition message, and the messages in the first message that satisfy the sampling rules are any one or more messages in the first partition message.

[0040] Specifically, all trace data is sent to the distributed message queue Kafka. The first consumer in the real-time Kafka consumer group pulls the first message, which is the first partition message. The first partition message contains multiple messages, and each message is processed for matching according to sampling rules. The message that matches the sampling rules can be any one or more messages in the first partition message. In addition, the traceId of each message that matches the sampling rules is written to a Redis Bloom filter.

[0041] In one embodiment, S304 further includes: storing the first message as hot data after the sampling operation is completed.

[0042] Specifically, after the sampling operation is completed, all data including the first message will be fully stored in storage as hot data. The TTL (Time To Live) for storage retention can be set to one hour.

[0043] In one example, the first consumer consumes end-to-end data from the distributed message queue Kafka in real time, writes the hot data to the full database for storage, and writes the traceId of the end-to-end trace data that needs to be retained to a Bloom filter according to cleaning rules. These cleaning rules include rules for extracting the most valuable trace data, such as end-to-end trace data with errors, time-consuming data, or trace data from specific APIs of certain apps.

[0044] After edge computing data cleaning, 1 to 2 hours of full hotspot data are retained, and this data is kept for more than 3 days. Because the value of end-to-end tracking data decreases over time, the more real-time the data, the greater its value. Therefore, real-time full hotspot data is retained for 1 to 2 hours.

[0045] S306, control the second consumer to pull the second message from the distributed message queue, compare the latest time of the second message with the latest time of the first message in the cache, and when it is determined from the comparison result that the latest time of the second message is delayed by a predetermined time value compared with the latest time of the first message, read the first link identification information from the storage, and when it is determined from the first link identification information that the second message needs to be sampled, treat the second message as valid full-link tracing data for cold data processing.

[0046] In this embodiment, a second consumer is set up for the distributed message queue. The second consumer, relative to the first consumer, consumes messages from the distributed message queue at a later time. A delay time is set, calculated from the moment the first consumer retrieves the first message from the distributed message queue. After the delay time is reached, the second consumer retrieves the second message from the distributed message queue. When the second consumer retrieves the second message, it obtains the timestamp of the second message to get its latest time. Then, the latest time of the second message is compared with the latest time of the first message in the cache. If the latest time of the second message is delayed by a predetermined time value compared to the latest time of the first message, the first link identification information is read from storage. If it is determined that the second message requires sampling based on the first link identification information, the second message is determined to be valid end-to-end tracking data and is processed as cold data. Cold data processing can involve storing the second message in a manner similar to cold data storage, thus saving the end-to-end tracking data.

[0047] In one embodiment, the predetermined time value is the time value for responding to the end-to-end request.

[0048] Specifically, the predetermined time value is set to the time required for the system to respond to the full-link request. Alternatively, it can be set to any time value greater than the time required to respond to the full-link request. When the predetermined time value is the time required to respond to the full-link request, the system has completed the response to the full-link request. That is, when the latest time of the second message is delayed by the predetermined time value compared to the latest time of the first message, it can be determined that the system has completed the response to the full-link request. Furthermore, the first link identification information is read from storage, and the second message that needs to be sampled is extracted using the first link identification information. The extracted second message is relatively complete compared to the full-link tracing, meaning that as complete as possible of the full-link tracing data can be extracted.

[0049] For example, a normal end-to-end request completes within 10 minutes, so after a 10-minute delay, the second consumer can retain the complete trace data of the sampled end-to-end request. Specifically, a relative delay of 10 minutes is achieved by determining that the message latency of each partition in all Kafka partitions relative to the first consumer consuming in real time is greater than 10 minutes.

[0050] In one embodiment, the second message is a second partition message. When it is determined that the second message needs to be sampled based on the first link identification information, the second message is treated as cold data for end-to-end tracking data. This includes: obtaining the second link identification information of any message in the second partition message; when the second link identification information is the same as the first link identification information, determining that any message needs to be sampled, and treating any message as cold data for end-to-end tracking data.

[0051] Specifically, a complete end-to-end tracing data set contains multiple segments. Each segment can be understood as a set of operations within an app that process requests, such as MySQL / REDIS operations. Each segment of the same complete end-to-end tracing data set contains the same trace ID. Therefore, when the second link identifier information is the same as the first link identifier information, it can be determined that the first message and the second message belong to the same end-to-end tracing data set. Thus, it can be determined that the corresponding message needs to be sampled and treated as cold data.

[0052] Therefore, valuable data after edge computing cleaning, such as data from faulty links, time-consuming links, core links, and other data that meet specific sampling rules, are highly valuable for observing performance bottlenecks and troubleshooting, and should be retained for at least 3 days. The purpose of the aforementioned cold and hot storage is to retain the maximum amount of valid data while minimizing storage costs.

[0053] Therefore, the first consumer consumes the first message in the distributed message queue in real time and records the first link identification information of messages that meet the sampling rules. When the second consumer consumes the second message in the distributed message queue, if the latest time of the second message consumed by the second consumer is delayed by a predetermined time compared to the message consumed by the first consumer (i.e., the latest time of the second message is delayed by a predetermined time value compared to the latest time of the first message), then the recorded first link identification information is used to determine whether the second message needs to be sampled. If so, the second message is treated as cold data. Thus, by having the first consumer identify messages that meet the sampling rules in real time, and then by setting a predetermined delay value, and then having the second consumer extract the messages required for end-to-end tracing that meet the sampling rules from the messages in the distributed message queue, more and more valuable end-to-end tracing data can be obtained, maximizing the value of the end-to-end system.

[0054] Regarding the data processing method for the above-mentioned end-to-end tracing, the following is a specific application scenario:

[0055] like Figure 4 As shown, the first consumer can specifically be a real-time Kafka consumer group, i.e. Figure 4 The RT C-Group in the context. The second consumer can specifically be a delayed Kafka consumer group, i.e. Figure 4 The Delay C-Group in the code. Bloom Filter is a Bloom filter. The basic processing flow of a full-link tracing data processing method includes:

[0056] 1. RT C-Group continuously consumes Kafka messages in real time.

[0057] 2.1 All hot data consumed by RT C-Group is stored in ES (Real-Time Search Engine) and HBase (a distributed, column-oriented open-source database).

[0058] 2.2 Record the data into a Bloom Filter according to the sampling rules for use in tail calculations.

[0059] 3. Delay C-Group delays consumption of Kafka messages by ten minutes.

[0060] 4. Filter out the valid trace data using a Bloom filter. The valid trace data is the data from the second message mentioned above. Store it in Elasticsearch and HBase.

[0061] For the specific application scenarios mentioned above, the following explanations are provided:

[0062] I. Complete Trace Data Description: A complete end-to-end trace contains multiple segments. Each segment can be understood as a set of operations that process requests within an app, such as MySQL / Redis operations. Each segment in the same complete trace contains the same traceId.

[0063] II. Explanation of core principles: The principle of trace data cleaning is to sample and match each operation in each segment of a complete trace chain. The sampling rules include erroneous operations, time-consuming operations, and operations of certain important apps. The traceId of the trace data that matches the sampling rules is written into the Redis Bloom filter.

[0064] III. Real-time Kafka Consumer Group Logic Description: All trace data is sent to Kafka. The real-time Kafka consumer group, upon receiving the first message, writes the latest message time from the first message of each partition to Redis (this is used for delayed processing by the delayed Kafka consumer group). Each first message is processed, a sampling operation is performed, and the traceId matching the sampling rules is written to a Redis Bloom filter. Specifically, one minute of data from the first message of each partition is divided into a Bloom filter for storage, retaining 60 minutes of data. After the sampling operation, all data is stored as hot data, retaining its TTL and expiring after one hour. Furthermore, due to the extremely large volume of full-link data, the throughput requirements for the Bloom filter are extremely high. Therefore, data determined by the cleaning rules to need retention is not further validated by the Bloom filter.

[0065] IV. Logic Description of Delayed Kafka Consumer Groups: Delayed Kafka consumer groups consume the second message from Kafka 10 minutes later than real-time Kafka consumer groups. Specifically: After pulling the second message, group it by partition. Obtain the latest time in the second message of each partition. If the timestamp of the Kafka message at that time is more than 10 minutes later than the time the message was written to Redis in the real-time Kafka consumer group, then the message is processed. Processing logic: Read the Bloom filter to get the traceId, and determine whether the second message needs to be sampled. Only sampled data will be stored as cold data, and cold data is stored for 7 days. If the time difference is less than 10 minutes, consumption is paused using `KafkaConsumer.pause`, i.e., a 10-minute pause minus the delay time of the real-time Kafka consumer group, controlling the delay of the second message in each partition of the delayed Kafka consumer group to be within 10 minutes.

[0066] Therefore, after edge computing data cleaning, we retain 1 to 2 hours of full hot data and more than 3 days of cleaned cold data. This is because the value of end-to-end data decreases over time, and the more real-time the data, the greater its value. So we retain 1 to 2 hours of full real-time hot data, and retain valuable data after edge computing cleaning (error links, time-consuming links, core links, and other data that meet specific sampling rules, which are of great value for observing performance bottlenecks and troubleshooting). The purpose of cold and hot storage is to retain the maximum amount of effective data while minimizing storage costs.

[0067] In one embodiment, the present invention also provides a data processing apparatus for end-to-end tracking. For example... Figure 5 As shown, the device includes:

[0068] The sending module 502 is used to send all data to the distributed message queue in real time.

[0069] The first processing module 504 is used to control the first consumer to pull the first message from the distributed message queue in real time, write the latest time of the first message into the cache, perform sampling operation on the first message using preset sampling rules, and store the first link identification information of the messages in the first message that meet the sampling rules.

[0070] The second processing module 506 is used to control the second consumer to pull the second message from the distributed message queue, compare the latest time of the second message with the latest time of the first message in the cache, and when it is determined from the comparison result that the latest time of the second message is delayed by a predetermined time value compared with the latest time of the first message, read the first link identification information from the storage, and when it is determined from the first link identification information that the second message needs to be sampled, process the second message as valid full-link tracking data for cold data processing.

[0071] For specific limitations regarding the data processing device for end-to-end tracing, please refer to the limitations of the data processing method for end-to-end tracing described above, which will not be repeated here. Each module in the aforementioned data processing device for end-to-end tracing can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0072] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores all data related to end-to-end tracing. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a data processing method for end-to-end tracing.

[0073] This invention provides a computer-readable storage medium storing an application program. When executed by a processor, the program implements a data processing method for end-to-end tracing as described in any of the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium that stores or transmits information in a readable form by a device (e.g., a computer, a mobile phone), and can be a read-only memory, a disk, or an optical disk, etc.

[0074] Furthermore, the above provides a detailed description of the data processing method, apparatus, computer equipment, and storage medium for end-to-end tracking provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method for full link tracking, characterized by, The method comprises: sending full data to a distributed message queue in real time; controlling a first consumer to pull a first message from the distributed message queue in real time, writing the latest time of the first message into a cache, performing a sampling operation on the first message according to a preset sampling rule, and storing first link identifier information of a message in the first message that meets the sampling rule; controlling a second consumer to pull a second message from the distributed message queue, comparing the latest time of the second message with the latest time of the first message in the cache, and when it is determined according to the comparison result that the latest time of the second message is delayed from the latest time of the first message by a predetermined time value, reading the first link identifier information from the storage, and when it is determined according to the first link identifier information that the second message needs to be sampled, performing cold data processing on the second message as valid full-link tracking data.

2. The method of claim 1, wherein, The distributed message queue is a kafka message queue, and / or the preset sampling rule includes an error operation rule and / or a high-time-consuming operation rule.

3. The method of claim 1, wherein, The first link identifier information of the message in the first message that meets the sampling rule is stored by: writing the first link identifier information of the message in the first message that meets the sampling rule into a Bloom filter for storage.

4. The method of claim 1, wherein, The predetermined time value is a time value for responding to a full-link request.

5. The method of claim 1, wherein, The first message is a first partition message, and the message in the first message that meets the sampling rule is one or more messages in the first partition message.

6. The method of claim 1, wherein, The method further comprises: after the sampling operation is completed, storing the first message as hot data.

7. The method of claim 1, wherein, The second message is a second partition message, and when it is determined according to the first link identifier information that the second message needs to be sampled, performing cold data processing on the second message as valid full-link tracking data, comprises: obtaining second link identifier information of any message in the second partition message; when the second link identifier information is the same as the first link identifier information, it is determined that the any message needs to be sampled, and the any message is processed as valid full-link tracking data.

8. A data processing apparatus for full link tracing, characterized in that, The apparatus comprises: a sending module configured to send full data to a distributed message queue in real time; a first processing module configured to control a first consumer to pull a first message from the distributed message queue in real time, write the latest time of the first message into a cache, perform a sampling operation on the first message according to a preset sampling rule, and store first link identifier information of a message in the first message that meets the sampling rule; The second processing module is configured to control the second consumer to pull a second message from the distributed message queue, compare the latest time of the second message with the latest time of the first message in the cache, and when it is determined according to the comparison result that the latest time of the second message is delayed from the latest time of the first message by a predetermined time value, read the first link identification information from the storage, and when it is determined according to the first link identification information that the second message needs to be subjected to a sampling operation, perform cold data processing on the second message as valid full-link tracking data.

9. A computer device comprising a memory, a processor, and an application stored on the memory and executable on the processor, wherein, The processor implements the steps of the method of any one of claims 1 to 7 when executing the application program.

10. A computer readable storage medium having stored thereon an application program, the application program comprising the steps of: The application program, when executed by the processor, implements the steps of the method of any one of claims 1 to 7.

Images