Buried point data transmission and storage method and system based on Kafka

Abstract

The invention discloses a Kafka-based buried point data transmission and storage method and system. The method comprises the following steps: collecting and sending buried point data of a client; the server receives, processes and forwards; storing and managing Kafka data; and the Dores starts the Route Load to consume the Kafka data. According to the system architecture, the buried point data can be quickly processed and stored, real-time analysis and utilization of the data are realized, a large amount of buried point data can be easily handled, blockage and loss of the data in the transmission process are avoided, the system architecture has better expansibility, new modules or components can be conveniently added to realize new functions, and the system architecture is convenient to use. And large-scale reconstruction of the whole architecture is not needed.

Description

Technical Field

[0001] The present invention belongs to the field of Internet technology, and in particular relates to a Kafka-based point-of-use data transmission and storage method and system. Background Art

[0002] With the development of Internet business, enterprises have an increasing demand for the analysis of user behavior data. By collecting and analyzing user behavior data within applications, enterprises can better understand user preferences, optimize product functions and formulate marketing strategies. Traditional data processing methods cannot meet the large-scale and continuously growing demand for point-of-sale data. On the one hand, directly writing a large amount of point-of-sale data into the database in real time may have a huge impact on database performance, causing the database to respond slowly or even crash; on the other hand, simple data storage methods lack effective data management and processing mechanisms, which is not conducive to subsequent data analysis. Summary of the invention

[0003] The purpose of the present invention is to propose a Kafka-based point-of-use data transmission and storage method and system to solve the problems of limited data processing capability, poor real-time performance and poor scalability in the prior art.

[0004] In order to solve the above technical problems, the present invention provides a method for transmitting and storing buried data based on Kafka, which adopts the following technical solution:

[0005] A method for transmitting and storing buried data based on Kafka, comprising the following steps:

[0006] S1, client-side data collection and transmission;

[0007] S11, initialization of buried points

[0008] During the development of the client application, embed the tracking code library, which is responsible for initializing the tracking function, including generating a unique client identifier and setting network parameters for communicating with the server.

[0009] S12, User Behavior Monitoring and Data Collection

[0010] When the user operates in the client application, the tracking code starts to monitor the user's various behaviors and collect relevant data information;

[0011] S13, Data Formatting and Encryption

[0012] The collected data is formatted into a specific format and encrypted, especially the part involving user sensitive information, such as user ID;

[0013] S14, data transmission

[0014] The client sends the formatted and encrypted tracking data to the server through the network protocol;

[0015] S2, the server receives, processes and forwards to Kafka;

[0016] S21, server monitoring and receiving

[0017] The server runs a network monitoring program, monitors the specified port, and waits for the client's connection request. After receiving the request, the server receives the data and parses the network request. If the data is encrypted, the server first decrypts it.

[0018] S22, Data Verification and Preprocessing

[0019] The server verifies the decrypted tracking data, checks the integrity of the data, and performs preprocessing operations on valid data, such as adding server-side timestamps and classifying and marking the data.

[0020] S23, Kafka producer configuration and data sending

[0021] The server configures the Kafka producer client to send the verified and pre-processed tracking data to the specified Kafka topic;

[0022] S3, Kafka data storage and management;

[0023] S31, Kafka cluster architecture and topic creation

[0024] Create topics for storing tracking data in Kafka in advance. Create multiple topics based on different business types or applications. Configure multiple partitions for each topic. The number of partitions is determined based on the expected data volume and processing requirements.

[0025] S32, Data storage and partition management

[0026] When the server sends the tracking data to the Kafka topic, Kafka stores the data in the corresponding partition according to the configured partition strategy. Kafka stores the data persistently on the disk and ensures the availability of the data through the replication mechanism.

[0027] S33, message queue management and data buffering

[0028] When the client sends data faster than the downstream system (Doris) consumes data, Kafka, as a message queue, buffers and manages a large amount of embedded data, temporarily stores data, and avoids data loss;

[0029] S4, Doris starts Routine Load to consume Kafka data;

[0030] S41, Routine Load task configuration

[0031] In the Doris system, configure the Routine Load task, including specifying the Kafka topic name, consumer group name, data format, data processing rules, etc. to be consumed. The consumer group is used to achieve load balancing and data distribution among multiple consumers.

[0032] S42, Data Acquisition and Conversion

[0033] After the Routine Load task is started, it establishes a connection with Kafka and obtains tracking data from the corresponding partition according to the allocation mechanism of the consumer group;

[0034] S43, Data processing and cleaning

[0035] The Doris system further processes and cleans the acquired tracking data, including removing duplicate data, processing missing values, and unifying data formats;

[0036] S44, data storage and index creation

[0037] The Doris system creates tables for storing tracking data based on business needs, and stores the processed and cleaned tracking data in the corresponding tables of the database.

[0038] Furthermore, in step S12, the relevant data information includes client identification, user behavior type, behavior-related object, timestamp, device information, etc.

[0039] Furthermore, in step S13, the data encryption algorithm adopts a common symmetric encryption algorithm or an asymmetric encryption algorithm, which is selected according to the security requirements of the system.

[0040] Furthermore, in step S14, during the data transmission process, the client will handle network connection abnormalities, such as automatic reconnection, setting a timeout, etc.

[0041] Furthermore, in step S22, the data integrity check includes whether all required fields exist and whether the data format meets expectations. If problems are found in the data, the server takes corresponding measures, such as recording error logs, trying to repair data or discarding invalid data.

[0042] Furthermore, in step S23, the server configures the Kafka producer client including specifying the address of the Kafka cluster (IP addresses and ports of multiple Kafka nodes), the Kafka topic name to which data is to be sent, and the like.

[0043] Furthermore, in step S31, the Kafka cluster is composed of multiple nodes (Brokers), and these nodes are interconnected through a network and work collaboratively.

[0044] Furthermore, in step S32, the partitioning strategy may be based on a certain field of the data (such as a hash value of a client identifier) ​​or a polling method.

[0045] Furthermore, in step S32, the data of each partition has copies on multiple nodes. When a node fails, the copies on other nodes can continue to provide data services.

[0046] Furthermore, in step S33, the Kafka provides an efficient message retrieval and order guarantee mechanism (within the same partition) to ensure that data is stored and consumed in the order in which it is sent.

[0047] Furthermore, in step S42, the Routine Load task converts the data from the format stored in Kafka to a format that can be processed by Doris during the process of acquiring data, and parses the data to convert the data in JSON format (or other formats) into a table structure in Doris.

[0048] Furthermore, in step S43, the deduplication of data is performed to determine whether the data is duplicated based on a specific unique identification field, such as a combination of a client identification, a timestamp, and a behavior type.

[0049] Further, in step S43, the processing of missing values ​​fills or marks missing values ​​according to business rules.

[0050] Furthermore, in step S44, the table structure includes fields such as client identification, behavior type, behavior object, timestamp, device information, etc. The Doris system creates indexes for frequently queried fields to improve query efficiency.

[0051] In order to solve the above technical problems, the present invention also provides a Kafka-based point-of-use data transmission and storage system, which adopts the following technical solution:

[0052] A Kafka-based point-of-use data transmission and storage system, comprising:

[0053] Web Service collection server, Kafka cluster, Doris cluster, the Web Service collection server includes a data receiving module, a data processing module, and a Kafka producer module, and the Doris cluster includes a Routine Load module, a data processing unit, and a Doris storage unit.

[0054] The beneficial effects of the present invention are as follows: by allowing the server to send the buried data to Kafka after receiving it, Doris starts the consumer task to consume the data in Kafka in real time, so that the buried data can be quickly processed and stored, realizing real-time analysis and utilization of the data; Kafka, as a distributed message queue, has high throughput and good concurrent processing capabilities, and can easily cope with a large amount of buried data, avoiding blockage and loss of data during transmission; the system architecture of the present invention has better scalability, and new modules or components can be easily added to realize new functions without large-scale reconstruction of the entire architecture.

[0055] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of the flow of the Kafka-based data transmission and storage method provided by the present invention;

[0057] Figure 2 This is a schematic diagram of the Kafka-based point-of-sale data transmission and storage system module provided by the present invention. DETAILED DESCRIPTION

[0058] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0059] refer to Figure 1 The present invention provides a method for transmitting and storing buried data based on Kafka, comprising the following steps:

[0060] S1, client-side data collection and transmission;

[0061] S11, initialization of tracking points

[0062] During the development of client applications (such as mobile apps or web apps), embed a tracking code library. This code library is responsible for initializing the tracking function, including generating a unique client identifier (if it does not exist yet), setting network parameters for communicating with the server (such as server address and port number), etc. For example, in a travel booking app, when a user opens the app for the first time, the tracking code library generates a specific device identifier as the client identifier and configures the network path for sending data to the server.

[0063] S12, User Behavior Monitoring and Data Collection

[0064] When the user operates in the client application, the tracking code starts to work. It monitors various user behaviors, such as clicking the booking button, browsing the tourist attraction page, modifying the search conditions, etc. For each detected behavior, the tracking code collects relevant data information, including client ID, user behavior type (such as click, browse, input, etc.), behavior-related objects (such as the ID of the booked hotel, the name of the browsed attraction), timestamp (accurate to milliseconds, recording the time when the behavior occurred), device information (such as device model, operating system version), etc. For example, when a user clicks the button to book a hotel, the tracking code collects data such as the client ID, the "click to book" behavior type, the hotel ID, the current timestamp, and the model of the mobile phone and the operating system version used by the user;

[0065] S13, Data Formatting and Encryption

[0066] The collected data will be formatted into a specific format, such as JSON format, which can facilitate data transmission in the network and subsequent parsing. At the same time, in order to ensure the security of the data, the data can be encrypted (especially the part involving user sensitive information, such as user ID). The encryption algorithm can use common symmetric encryption or asymmetric encryption algorithm, which is selected according to the security requirements of the system. For example, the collected data {"client_id":"abc123","action":"book_hotel","hotel_id":"456","timestamp":"2024 - 11 - 07 14:30:00","device_info":{"model":"iPhone 14","os_version":"iOS 16"}} is encrypted to form an encrypted data stream;

[0067] S14, data transmission

[0068] The client sends the formatted (and encrypted) tracking data to the server through a network protocol (such as an HTTP POST request). During the sending process, the client will handle network connection anomalies, such as automatic reconnection and setting timeouts. For example, if the first data sending fails, the client will try to reconnect to the server and send data after a certain time interval (such as 5 seconds) until the maximum number of retries (such as 3 times) is reached.

[0069] S2, the server receives, processes and forwards to Kafka;

[0070] S21, server monitoring and receiving

[0071] The server runs a network listening program, listens to the specified port, and waits for the client's connection request. After receiving the tracking data request sent by the client, the server receives the data and parses the network request. If the data is encrypted, the server first decrypts it using the corresponding decryption key. For example, the server receives an HTTP POST request and parses the request body to obtain the tracking data content.

[0072] S22, Data Verification and Preprocessing

[0073] Verify the decrypted tracking data and check the integrity of the data, including whether all required fields exist (such as client ID, behavior type), whether the data format meets expectations (such as whether the timestamp format is correct). If problems are found in the data, the server can take corresponding measures, such as recording error logs, trying to repair the data (if possible), or discarding invalid data. For valid data, the server can perform some pre-processing operations, such as adding server-side timestamps, classifying and marking data, etc. For example, if the received timestamp format is incorrect, the server tries to repair it according to the current server time. If the data is complete and valid, the server adds the timestamp of the received data.

[0074] S23, Kafka producer configuration and data sending

[0075] The server configures the Kafka producer client, which includes specifying the address of the Kafka cluster (IP addresses and ports of multiple Kafka nodes), the name of the Kafka topic to send data, etc., and sends the verified and pre-processed tracking data to the specified Kafka topic. For example, the server sends all tracking data to a Kafka topic named "Travel Booking Tracking Data Topic", which is specifically used to store tracking data related to the travel booking app;

[0076] S3, Kafka data storage and management;

[0077] S31, Kafka cluster architecture and topic creation

[0078] The Kafka cluster consists of multiple nodes (Brokers), which are interconnected through the network and work together. Topics for storing tracking data are created in advance in Kafka. Multiple topics can be created according to different business types or applications. Each topic can be configured with multiple partitions. The number of partitions is determined based on the expected data volume and processing requirements. For example, for the tracking data topic of a travel booking APP, data can be allocated to different partitions according to different business modules (such as hotel reservations, attraction ticket reservations, transportation reservations, etc.) to improve the parallelism of data processing;

[0079] S32, Data storage and partition management

[0080] After the server sends the tracking data to the Kafka topic, Kafka stores the data in the corresponding partition according to the configured partition strategy. The partition strategy can be based on a certain field of the data (such as the hash value of the client identifier) ​​or polling. Kafka stores the data persistently on the disk and ensures the availability of the data through the replication mechanism. The data of each partition has replicas on multiple nodes. When a node fails, the replicas on other nodes can continue to provide data services. For example, if the partition strategy based on the client identifier hash value is adopted, the tracking data of the same client is usually stored in the same partition, which is convenient for subsequent query and processing;

[0081] S33, message queue management and data buffering

[0082] As a message queue, Kafka buffers and manages a large amount of embedded data. It can handle data production and consumption at different rates. When the client sends data faster than the downstream system (Doris) consumes data, Kafka can temporarily store the data to avoid data loss. At the same time, Kafka provides an efficient message retrieval and order guarantee mechanism (within the same partition) to ensure that data is stored and consumed in the order it is sent. For example, during the peak period of travel booking apps, a large number of users operate at the same time to generate a large amount of embedded data. Kafka can stably store and manage this data and wait for Doris to consume it.

[0083] S4, Doris starts Routine Load to consume Kafka data;

[0084] S41, Routine Load task configuration

[0085] In the Doris system, configure the Routine Load task, which includes specifying the Kafka topic name to be consumed (consistent with the topic to which the server sends data), the consumer group name, the data format (such as JSON), data processing rules, etc. The consumer group is used to achieve load balancing and data distribution among multiple consumers. For example, configure a consumer group named "Travel Booking Data Consumption Group" to consume data in the "Data Consumption Topic - Travel Booking" and set data processing rules, such as type conversion for certain fields;

[0086] S42, Data Acquisition and Conversion

[0087] After the Routine Load task is started, a connection is established with Kafka to obtain the buried data from the corresponding partition according to the allocation mechanism of the consumer group. In the process of obtaining data, the data is converted from the format stored in Kafka to a format that can be processed by Doris. At the same time, the data is parsed and the data in JSON format (or other formats) is converted into a table structure in Doris. For example, data such as {"client_id":"abc123","action":"book_hotel","hotel_id":"456","timestamp":"2024-11-0714:30:00","device_info":{"model":"iPhone 14","os_version":"iOS 16"}} is converted into a row of data in the Doris table, and each field corresponds to a column in the table;

[0088] S43, Data processing and cleaning

[0089] The acquired tracking data is further processed and cleaned, including removing duplicate data (determining whether it is duplicated based on a combination of specific unique identification fields, such as client ID, timestamp, and behavior type), processing missing values ​​(filling or marking missing values ​​based on business rules), and unifying data formats (such as unifying timestamps of different formats into a standard format). For example, if two data are found to have a slight difference in timestamps (possibly due to repeated transmission caused by network delays) and other key information is exactly the same, one of the data is deleted; if the device model field in a certain data is missing, it is obtained from other data sources based on the client ID or marked as "unknown";

[0090] S44, data storage and index creation

[0091] The processed and cleaned tracking data are stored in the corresponding tables of the Doris database. In Doris, tables for storing tracking data are created in advance according to business needs. The table structure includes fields such as client ID, behavior type, behavior object, timestamp, device information, etc., and indexes are created for frequently queried fields (such as timestamp, behavior type) to improve query efficiency. For example, the processed travel booking APP tracking data is stored in a Doris table named "Travel Booking Tracking Data Table", and a B-tree index is created for the "timestamp" field to facilitate subsequent queries on user behavior data based on time ranges.

Claims

1. A method for transmitting and storing buried data based on Kafka, characterized in that: The steps include: S1, client-side data collection and transmission; S11, during the client application development process, embed a tracking code library, the tracking code library initializes tracking, generates a unique client identifier, and sets network parameters for communication with the server; S12, when the user operates in the client application, the tracking code starts to monitor various behaviors of the user and collect relevant tracking data information; S13, the buried point data is formatted into a specific format and data encryption is performed; S14, the client sends the formatted and encrypted tracking data to the server through a network protocol; S2, server receives, processes and forwards; S21, the server runs a network monitoring program, monitors a designated port, and waits for a connection request from the client. After receiving the connection request, the server receives the buried data, decrypts it, and parses the network request; S22, the server verifies the decrypted embedding data, checks the data integrity, and pre-processes the valid data; S23, the server configures the Kafka producer client to send the verified and pre-processed tracking data to the specified Kafka topic; S3, Kafka data storage and management; S31, build a Kafka cluster, create a topic for storing the tracking data in advance in the Kafka, create multiple topics according to different business types and applications, and configure multiple partitions for each topic; S32, when the server sends the tracking data to the Kafka topic, the Kafka stores the data in the corresponding partition according to the configured partition strategy, and the Kafka stores the tracking data persistently on the disk, while ensuring the availability of the data through a replica mechanism; S33, when the speed at which the client sends data is faster than the speed at which the downstream system consumes data, the Kafka acts as a message queue to temporarily store a large amount of the embedded point data for buffering; S4, Doris starts Routine Load to consume Kafka data; S41, in the Doris system, configure the Routine Load task, including specifying the Kafka topic name, consumer group name, data format, and data processing rules to be consumed. The consumer group is used to achieve load balancing and data distribution among multiple consumers; S42, after the Routine Load task is started, a connection is established with the Kafka, and the buried point data is obtained from the corresponding partition according to the allocation mechanism of the consumer group; S43, the Doris system further processes and cleans the acquired buried point data; S44, the Doris system creates a table for storing the buried data according to business needs, and stores the processed and cleaned buried data in the table of the database.

2. The method for transmitting and storing buried data based on Kafka according to claim 1 is characterized in that: In step S12, the buried data information includes client identification, user behavior type, behavior-related objects, timestamp, and device information.

3. The method for transmitting and storing buried data based on Kafka according to claim 1 is characterized in that: In step S14, during the process of sending the buried data, the client will handle network connection abnormalities.

4. The method for transmitting and storing buried data based on Kafka according to claim 1 is characterized in that: In step S31, the Kafka cluster consists of multiple nodes.

5. The method for transmitting and storing buried data based on Kafka according to claim 1 is characterized in that: In step S32, the tracking data of each partition has copies on multiple nodes. When a node fails, the copies on other nodes continue to provide data services.

6. The method for transmitting and storing buried data based on Kafka according to claim 5 is characterized in that: In the step S32, the partitioning strategy is based on a field of data and a polling method.

7. The method for transmitting and storing buried data based on Kafka according to claim 1 is characterized in that: In the step S42, the Routine Load task converts the buried data from the format stored in the Kafka into a format that can be processed by Doris during the process of acquiring the buried data.

8. The method for transmitting and storing buried data based on Kafka according to claim 1 is characterized in that: In step S43, the Doris system further processes and cleans the acquired buried data including removing duplicate data, and the Doris system determines whether the buried data is duplicated based on a specific unique identification field.

9. The method for transmitting and storing buried data based on Kafka according to claim 1 is characterized in that: In the step S44, the table used to store the buried data has a table structure including client identification, behavior type, behavior object, timestamp, and device information fields, and the Doris system creates indexes for frequently queried fields.

10. A Kafka-based point-of-use data transmission and storage system, characterized in that: include: Web Service collection server, Kafka cluster, Doris cluster, the Web Service collection server includes a data receiving module, a data processing module, and a Kafka producer module, and the Doris cluster includes a Routine Load module, a data processing unit, and a Doris storage unit.

Images