A method, device, and medium for processing user trace data based on a messaging system

Abstract

The application discloses a user trace data processing method and device based on a message system and a medium, belongs to the technical field of Internet and information systems, and is used for solving the technical problem that a large number of use records and operation logs generated in the current information system are difficult to be collected, analyzed and processed in time, and it is difficult for managers and users to make intuitive and effective decisions, which easily affects the efficient use and mastery of the information system by personnel. The method comprises the following steps: obtaining user trace data through a preset interceptor; sending the user trace data to a Kafka topic based on Kafka Stream and storing the user trace data; performing data analysis on the user trace data according to a preset analysis model to obtain analyzed user trace data; and storing the analyzed user trace data in a preset time sequence database, so as to facilitate managers to make information decisions on the information system according to the analyzed user trace data.

Description

Technical Field

[0001] This application relates to the field of Internet and information systems, and in particular to a method, device and medium for processing user trace data based on a messaging system. Background Technology

[0002] In today's society, numerous platform systems, websites, and other information technology products have been built across various industries. Users generate a series of usage records or operation logs when using these systems. This data can be called user footprints. User footprint data can provide relevant data analysis for our product design iterations and can also serve as a basis for performance evaluation in certain scenarios.

[0003] Kafka is a high-throughput distributed publish-subscribe messaging system that can handle streaming data from all consumer actions on a website. Kafka Streams is a new feature introduced in Apache Kafka version 0.10. It provides the capability for streaming computation and analysis of data stored within Kafka. InfluxDB (TSDB) is a time-series database designed to handle high write and query loads. TSDB is a database optimized for timestamped or time-series data, specifically built for processing timestamped metrics and events or metrics.

[0004] Currently, the collection and analysis of the large amount of usage records and operation logs generated by information systems is slow. It is difficult for managers or users to make real-time decisions and adjustments based on changes in usage records and operation logs, resulting in a certain degree of information processing lag and making it difficult to improve work efficiency. Summary of the Invention

[0005] This application provides a user trace data processing method, device, and medium based on a messaging system to solve the following technical problem: Currently, the large amount of usage records and operation logs generated in information systems are difficult to collect, analyze, and process in a timely manner, making it difficult for managers and users to make intuitive and effective decisions, which can easily affect the efficient use and mastery of information systems by personnel.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] On one hand, embodiments of this application provide a method for processing user trace data based on a messaging system. The method includes: acquiring user trace data through a preset interceptor; sending the user trace data to a Kafka topic based on KafkaStream and storing the user trace data; parsing the user trace data according to a preset parsing model to obtain parsed user trace data; and storing the parsed user trace data in a preset time-series database to facilitate management personnel in making information decisions about the information system based on the parsed user trace data.

[0008] This application embodiment processes user trace data based on Kafka Stream and the time-series database InfluxDB. First, it maintains long-running tasks through streaming computing, ensuring that each Kafka topic runs continuously and can continuously receive a large number of usage records and operation logs from the information system. It can also achieve rapid collection, analysis, and processing. Through rich data parsing models, it can quickly parse and receive user trace data, and enable information system administrators or users to more intuitively and efficiently grasp the system usage and make some decisions.

[0009] In one feasible implementation, user trace data is obtained by pre-setting an interceptor, specifically including: receiving various types of user trace data under the same standard to obtain a unified reception standard; determining the interceptor according to the unified reception standard; wherein, the interceptor is used to intercept non-user trace data of various services in the information system; filtering the data of various services through the interceptor, and collecting the filtered user trace data to obtain the user trace data of the information system.

[0010] In one feasible implementation, before sending the user trace data to a Kafka topic based on Kafka Stream and storing the user trace data, the method further includes: determining a Kafka topic and a Kafka data stream that match the user trace data based on the relevant functions of Kafka Stream; wherein, Kafka Stream provides functions for streaming computation and analysis of data stored in Kafka, and the relevant functions of Kafka Stream include at least: application state management functions, fast and efficient aggregation and joining functions, functions for correctly distinguishing event time and processing time, and functions for seamlessly handling late and out-of-order data; binding the Kafka topic to the Kafka data stream to obtain the data binding relationship between the Kafka topic and the Kafka data stream; wherein, the Kafka topic is used to specify the direction of the Kafka data stream.

[0011] In one feasible implementation, the user trace data is sent to a Kafka topic based on Kafka Streams and stored, specifically including: starting a preset user trace data receiver; initializing the Kafka data stream to obtain an initialized Kafka data stream; obtaining the user trace data from the interceptor through the user trace data receiver; establishing a one-to-one correspondence between the user trace data and the initialized Kafka data stream; inputting the user trace data into the Kafka topic corresponding to the initialized Kafka data stream according to the data binding relationship between the Kafka topic and the Kafka data stream; and storing the user trace data in the corresponding Kafka topic.

[0012] In one feasible implementation, after storing the user trace data in the corresponding Kafka topic, the method further includes: continuously sending the Kafka topic to the corresponding user through streaming computation, so that the user can update the user trace data in the Kafka topic in real time to obtain an updated Kafka topic.

[0013] This application embodiment maintains long tasks through streaming computing, ensuring that each Kafka topic runs continuously and receives data from the information system.

[0014] In one feasible implementation, before parsing the user trace data according to a preset parsing model to obtain parsed user trace data, the method further includes: obtaining the data association relationship between the user trace data and different business function URI interfaces in the information system; and constructing a model for parsing the user trace data under different business functions according to the data association relationship to obtain the parsing model.

[0015] This application defines a data parsing model through data association or custom configuration. By using a rich set of data parsing models to parse various types of user trace data, the development process for connecting to various types of user trace data is simplified, and the received user trace data can be parsed quickly.

[0016] In one feasible implementation, the user trace data is parsed according to a preset parsing model to obtain parsed user trace data. Specifically, this includes: starting a preset user trace data parser; wherein the user trace data parser is used to monitor updated Kafka topics; if the updated Kafka topic is detected to have updated user trace data, then a parsing model corresponding to the updated user trace data is determined based on the data association relationship corresponding to the updated user trace data; the updated user trace data is parsed using the parsing model to obtain the parsed user trace data; wherein the parsed user trace data includes at least: time measurement information and operation measurement information.

[0017] In one feasible implementation, the parsed user trace data is stored using a pre-set time-series database. Specifically, this includes: acquiring time measurement information from the parsed user trace data and establishing a data index for the parsed user trace data based on the time measurement information; storing the data index and the corresponding parsed user trace data in an InfluxDB time-series database to facilitate information decision-making by administrators based on the parsed user trace data; wherein the InfluxDB time-series database is used for tracking, monitoring, downsampling, and aggregating metrics or events over time, and the InfluxDB time-series database includes at least: a server metrics database, an application performance database, a network database, and a sensor database.

[0018] This application embodiment utilizes the time attribute of user trace data and specifically employs the time series database InfluxDB to maximize the value of the data.

[0019] Secondly, embodiments of this application also provide a user trace data processing device based on a messaging system, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to execute a user trace data processing method based on a messaging system as described in any of the above embodiments.

[0020] Thirdly, embodiments of this application also provide a non-volatile computer storage medium, characterized in that the storage medium is a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium stores at least one program, each program including instructions, and when the instructions are executed by a terminal, the terminal executes a user trace data processing method based on a message system as described in any of the above embodiments.

[0021] This application provides a method, device, and medium for processing user trace data based on a messaging system. It processes user trace data using Kafka Stream and the time-series database InfluxDB. First, it maintains long-running tasks through streaming computing, ensuring each Kafka topic runs continuously and receives a large amount of usage records and operation logs from the information system. It also enables rapid collection, analysis, and processing. Through rich data parsing models, it analyzes various types of user trace data, allowing for rapid parsing of received user trace data. Furthermore, it enables information system administrators or users to more intuitively and efficiently grasp system usage and make informed decisions. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0023] Figure 1 A flowchart of a user trace data processing method based on a messaging system is provided for embodiments of this application;

[0024] Figure 2 This application provides a schematic diagram of the overall process for processing user trace data.

[0025] Figure 3 This is a schematic diagram of the structure of a user trace data processing device based on a messaging system, provided in an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0027] This application provides a method for processing user trace data based on a messaging system, such as... Figure 1 As shown, the user trace data processing method based on the messaging system specifically includes steps S101-S104:

[0028] S101. Obtain user trace data through a preset interceptor.

[0029] Specifically, various types of user log data are received under the same standard to obtain a unified reception standard. Based on this unified reception standard, an interceptor is determined. This interceptor is used to block non-user log data from various business processes within the information system.

[0030] Furthermore, the interceptor filters the data from various business processes and collects the filtered user trace data to obtain the user trace data of the information system.

[0031] In one embodiment, Figure 2 This application provides a schematic diagram of the overall process for processing user trace data, as shown in the embodiments. Figure 2 As shown, interceptors are used to obtain trace data from several users in the business system. The interceptors are defined based on the same data reception standard.

[0032] S102. Send the user trace data to a Kafka topic based on Kafka Stream and store the user trace data.

[0033] Specifically, based on the relevant functions of Kafka Streams, Kafka topics and Kafka data streams that match the user's trace data are determined. Among them, Kafka Streams provides the ability to perform streaming computation and analysis on data stored in Kafka. The relevant functions of Kafka Streams include at least: application state management, fast and efficient aggregation and joining, accurate differentiation between event time and processing time, and seamless handling of late and out-of-order data.

[0034] Furthermore, the Kafka topics are bound to the Kafka data streams to establish a data binding relationship between them. The Kafka topic defines the direction of the Kafka data stream.

[0035] Furthermore, the preset user trace data receiver is started. The Kafka data stream is then initialized to obtain the initialized Kafka data stream.

[0036] In one embodiment, the user trace data receiver primarily has the following functions: defining a unified user trace data interceptor; defining a Kafka topic for pushing user trace data; defining a data stream KStream, i.e., a Kafka data stream, based on the functionality of Kafka Streams, to specify the direction of Kafka data flow; and sending the received data to the corresponding Kafka topic.

[0037] Furthermore, user trace data in the interceptor is obtained through the user trace data receiver.

[0038] Furthermore, a one-to-one mapping is established between user trace data and the initial Kafka data stream. Based on the data binding relationship between Kafka topics and Kafka data streams, user trace data is input into the Kafka topic corresponding to the initial Kafka data stream. The user trace data is then stored in the corresponding Kafka topic.

[0039] Furthermore, through streaming computation, Kafka topics are continuously sent to the corresponding users, enabling users to update the user trace data in the Kafka topics in real time and obtain updated Kafka topics.

[0040] In one embodiment, such as Figure 2 As shown, after the user trace data receiver starts, it initializes the Kafka data stream and binds the user trace data to Kafka topics. Simultaneously, it intercepts interface requests from various business functions within the information system, which continuously pushes data to the user trace data receiver. Upon receiving data, the user trace data receiver, according to the binding relationship between the Kafka data stream and Kafka topics, sends the data to the corresponding Kafka topic. Through stream processing, it continuously sends the Kafka topic to the corresponding user, enabling the user to update the user trace data in the Kafka topic in real time and obtain the updated Kafka topic.

[0041] S103. Based on the preset parsing model, perform data parsing on the user trace data to obtain the parsed user trace data.

[0042] Specifically, the data association between user trace data and the URI interfaces of different business functions in the information system is obtained. Based on the data association, a model is constructed by parsing the user trace data under different business functions to obtain a parsed model.

[0043] In one embodiment, the operation function paths, names, etc. of the information system first need to be associated with the user trace data information and the corresponding different business function URI interfaces. Based on the association relationship, a parsing model is constructed for the user trace data under various business functions.

[0044] In one embodiment, the user log data parser parses various types of user log data, from simple to complex, by defining parsing models for each type. The parsing models also support custom configuration construction.

[0045] Furthermore, the default user trace data parser is started. This parser is used to listen for updates to the Kafka topic.

[0046] Furthermore, if an update to user trace data is detected on a Kafka topic, the parsing model corresponding to the updated user trace data is determined based on the data relationships associated with the updated user trace data. The user trace data parser can listen for updates to Kafka topics using a single service or multiple services.

[0047] Furthermore, the updated user trace data is parsed using a analytical model to obtain parsed user trace data. This parsed user trace data includes at least: time measurement information and operation measurement information.

[0048] In one embodiment, such as Figure 2 As shown, the user trace data parser defines corresponding parsing models and storage for various types of user trace data. After starting, the user trace data parser begins listening for all updated Kafka topics. When it receives updated user trace data from an updated Kafka topic, the user trace data parser retrieves the corresponding parsing model and then parses the data in the updated user trace data to obtain the parsed user trace data.

[0049] S104. By pre-setting a time-series database, the parsed user trace data is stored to facilitate management personnel in making information decisions about the information system based on the parsed user trace data.

[0050] Specifically, the time measurement information in the parsed user trace data is obtained, and a data index for the parsed user trace data is established based on the time measurement information.

[0051] Furthermore, based on the InfluxDB time-series database, data indexes and corresponding parsed user trace data are stored to facilitate information decision-making by administrators based on the parsed user trace data. The InfluxDB time-series database is used to track, monitor, downsample, and aggregate metrics or events over time, and includes at least: a server metrics database, an application performance database, a network database, and a sensor database.

[0052] In one embodiment, such as Figure 2 As shown, the user trace data parser uses the InfluxDB time series database storage method to store the parsed user trace data, making full use of the time measurement inherent in the user trace data itself, and thus tracking the data over time. In other words, by using the InfluxDB time series database to maximize the value of the data, the time attribute of the parsed user trace data can be maximized.

[0053] In one embodiment, a series of user trace data such as usage records or operation logs generated by the information system are processed and parsed by Kafka Stream to obtain parsed user trace data. The data index and the parsed user trace data corresponding to the data index are stored in the InfluxDB time series database, which is indexed by time.

[0054] In addition, embodiments of this application also provide a user trace data processing device based on a messaging system, such as... Figure 3 As shown, the user trace data processing device 300 based on the messaging system specifically includes:

[0055] At least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions executable by the at least one processor 301 to enable the at least one processor 301 to execute:

[0056] User data is captured by pre-set interceptors;

[0057] Send user trace data to a Kafka topic based on Kafka Streams and store the user trace data;

[0058] Based on the preset parsing model, the user trace data is parsed to obtain the parsed user trace data.

[0059] By pre-setting a time-series database, the parsed user trace data is stored to facilitate management personnel in making information-based decisions based on the parsed user trace data.

[0060] This application provides a method, device, and medium for processing user trace data based on a messaging system. It processes user trace data using Kafka Stream and the time-series database InfluxDB. First, it maintains long-running tasks through streaming computing, ensuring each Kafka topic runs continuously and receives a large amount of usage records and operation logs from the information system. It also enables rapid collection, analysis, and processing. Through rich data parsing models, it analyzes various types of user trace data, allowing for rapid parsing of received user trace data. Furthermore, it enables information system administrators or users to more intuitively and efficiently grasp system usage and make informed decisions.

[0061] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0062] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0063] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0066] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0067] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0068] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0069] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] The above description is merely an embodiment of this application and is not intended to limit this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of this application should be included within the scope of the claims of this application.

Claims

1. A message system-based user footprint data processing method, characterized by, The method comprises: acquiring user trace data through a preset interceptor; determining a Kafka topic and a Kafka data stream that meet the user trace data based on the functions of Kafka Stream, wherein the Kafka Stream provides functions of stream computing and analysis on data stored in Kafka, and the functions of Kafka Stream at least include application state management, fast and effective aggregation and connection, correct distinction between event time and processing time, and seamless processing of late data and unordered data; binding the Kafka topic and the Kafka data stream in data correspondence to obtain a data binding relationship between the Kafka topic and the Kafka data stream, wherein the Kafka topic is used to define the flow direction of the Kafka data stream; sending the user trace data to a Kafka topic based on Kafka Stream and storing the user trace data, specifically comprising: starting a preset user trace data receiver and initializing the Kafka data stream to obtain an initialized Kafka data stream; acquiring the user trace data in the interceptor through the user trace data receiver; performing one-to-one data correspondence between the user trace data and the initialized Kafka data stream; inputting the user trace data into the Kafka topic corresponding to the initialized Kafka data stream according to the data binding relationship between the Kafka topic and the Kafka data stream; storing the user trace data in the Kafka topic in correspondence, and continuously sending the Kafka topic to the corresponding user through stream computing, so that the user updates the user trace data in the Kafka topic in real time to obtain an updated Kafka topic; performing data analysis on the user trace data according to a preset analysis model to obtain analyzed user trace data; storing the analyzed user trace data in a preset time series database to facilitate information decision-making on the information system by management personnel according to the analyzed user trace data.

2. The user footprint data processing method based on a message system according to claim 1, characterized in that, acquiring user trace data through a preset interceptor, specifically comprising: receiving data of various types of user trace data under the same standard to obtain a unified receiving standard; determining the interceptor according to the unified receiving standard, wherein the interceptor is used to intercept non-user trace data of each business in the information system; filtering the data of each business through the interceptor, collecting the filtered user trace data, and obtaining the user trace data of the information system.

3. The user footprint data processing method based on a message system according to claim 1, characterized in that, Before performing data analysis on the user trace data according to a preset analysis model to obtain analyzed user trace data, the method further comprises: acquiring a data association relationship between the user trace data and different business function URI interfaces in the information system; According to the data association relationship, the user trace data under different service functions is parsed and constructed to obtain the analysis model.

4. The user footprint data processing method based on a message system according to claim 3, characterized in that, According to the preset analysis model, the user trace data is parsed to obtain parsed user trace data, specifically including: Starting a preset user trace data parser; wherein the user trace data parser is used to listen to an updated Kafka topic; If the updated Kafka topic is found to have updated user trace data, the analysis model corresponding to the updated user trace data is determined according to the data association relationship corresponding to the updated user trace data; Through the analysis model, the updated user trace data is parsed to obtain the parsed user trace data; wherein the parsed user trace data at least includes time measurement information and operation measurement information.

5. The user footprint data processing method based on a message system according to claim 1, characterized in that, Through a preset time sequence database, the parsed user trace data is stored, specifically including: Obtaining the time measurement information in the parsed user trace data, and establishing a data index of the parsed user trace data according to the time measurement information; According to the InfluxDB time sequence database, the data index and the parsed user trace data corresponding to the data index are stored to facilitate the management personnel to make information decisions for the information system according to the parsed user trace data; wherein the InfluxDB time sequence database is used to track, monitor, downsample and aggregate measurements or events over time, and the InfluxDB time sequence database at least includes a server index library, an application performance library, a network database and a sensor database.

6. A message system based user footprint data processing device, characterized by, The device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein The memory stores instructions executable by the at least one processor to enable the at least one processor to execute a user trace data processing method based on a message system according to any one of claims 1-5.

7. A non-transitory computer storage medium, comprising, The storage medium is a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores at least one program, and each program includes instructions which, when executed by a terminal, cause the terminal to execute a user trace data processing method based on a message system according to any one of claims 1-5.

Images