Data flow-based method and device for visual verification of a data flow-based method and a readable storage medium

Abstract

The application provides a data flow-based point burying visualization verification method and device and a readable storage medium thereof, acquires user APP interaction data through a point burying collection service, transmits the data to the collection service after compression and encoding, forwards the data to Kafka after analyzing and filtering abnormal data and adding attributes such as timestamps and geographic information, strictly matches and filters the data based on a broadcast state by using a streaming service, writes data meeting the conditions into a database, supports multi-dimensional screening such as time, device ID and event type through a visual interface, and realizes real-time preview and verification of point burying data. The application solves the problems of the prior art that point burying data cannot be previewed, real-time performance is poor and storage is not accurate, realizes data integrity verification, second-level delay preview and accurate filtering through data flow full-link optimization, and improves point burying development efficiency and reliability.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, specifically to the technology of collecting, processing, verifying and visualizing embedded data, and particularly to a data flow-based embedded data visualization verification method, apparatus and its readable storage medium. Background Technology

[0002] In existing technologies, when users interact with an app, event tracking data is reported by a data collection service, processed, and then forwarded to Kafka. A streaming service then reads the configuration file, consumes the data, and writes it to a MySQL database. Finally, microservices are used to display the data. The specific process includes: event tracking data is reported and processed by the data collection service; the processed data is forwarded to Kafka; the streaming service consumes the data and writes it to the database; and finally, microservices are used to implement page display and filtering functions.

[0003] However, existing technologies have significant drawbacks:

[0004] 1. Lack of data visualization: The data tracking lacks real-time preview capabilities during the collection, processing, and storage stages, making it difficult for the business side to quickly verify the effectiveness of the tracking.

[0005] 2. Inability to monitor in real time: The delay from data point reporting to data display is high (minutes or even longer), which leads to delayed anomaly detection and makes it impossible to detect data loss or format errors in a timely manner.

[0006] 3. Insufficient data accuracy: Existing solutions do not provide a dynamic filtering mechanism, making it impossible to accurately locate embedded data for specific devices or events, and lack integrity verification of private attributes.

[0007] Therefore, there is an urgent need for a data flow-based method, device, and readable storage medium for visual verification of data tracking points to solve the problems existing in the current technology. Summary of the Invention

[0008] This invention provides a data flow-based method, apparatus, and readable storage medium for visual verification of data tracking points. It addresses the problems in current technologies where the data tracking point collection and processing flow lacks real-time visualization, dynamic filtering, and integrity verification mechanisms, resulting in high risk of data loss, delayed anomaly detection, and low verification efficiency.

[0009] The core technology of this invention is a data flow-based visualization verification method for tracking data. This method involves collecting, compressing, encoding, streaming, and filtering device data based on broadcast JOIN, combined with real-time visualization filtering, to achieve integrity verification, real-time preview, and accurate data entry for tracking data.

[0010] In a first aspect, the present invention provides a data flow-based method for visual verification of event tracking, the method comprising the following steps:

[0011] User APP interaction data is collected through the event tracking service. The interaction data includes public attributes and private attributes. The public attributes are located in the first-level data structure, and the private attributes are located in the second-level data structure properties.

[0012] The collected data is compressed and encoded, and then transmitted to the data acquisition service.

[0013] The data collection service parses and processes the data, filters out abnormal data, adds timestamps, geographic information and personalized attributes, and then forwards it to Kafka;

[0014] Consume real-time data from Kafka using streaming services, filter the data by device ID based on broadcast status, and retain only data that matches the preset device ID;

[0015] The filtered data is written to the database, and a multi-dimensional filtering function is provided through a visual interface to enable real-time preview and verification of the data.

[0016] Furthermore, the data is compressed and encoded, including: compressing the data using gzip, encoding the compressed data using Base64, and encoding the URL.

[0017] Furthermore, the data acquisition service parses and processes the data, including: decoding and format verification of the encoded data layer by layer, discarding non-compliant data and logging it; obtaining the timestamp of the server's time zone and adding it to the data; and resolving the IP address from the IP address database to obtain geographical information and adding it to the data.

[0018] Furthermore, the streaming service is used to consume data from Kafka in real time, and the data is filtered by device ID based on the broadcast status. This includes configuring device IDs through the front-end interface and sending them to the broadcast stream, and using a strict matching mode to filter device IDs in the event tracking data. Data that does not meet the rules is discarded directly.

[0019] Furthermore, the streaming service uses the Flink or Spark Streaming engine to achieve JOIN processing of low-throughput device filtering data streams and high-throughput instrumentation data streams through broadcast states; the Kafka utility class is initialized using the singleton pattern, sets the number of data sending retries, and configures device filtering rules.

[0020] Furthermore, the visual interface supports filtering based on time, platform, device ID, event type, and attributes, and the returned data includes structured information such as event name, time, platform source, and attribute definition.

[0021] Furthermore, the filtered data is written to the database, including: using JdbcSink to write to the database, setting up a write retry mechanism and batch submission parameters.

[0022] Secondly, the present invention provides a data flow-based event tracking visualization verification device, comprising:

[0023] The event tracking module is used to collect user APP interaction data through the event tracking service. The interaction data includes public attributes and private attributes. The public attributes are located in the first-level data structure, and the private attributes are located in the second-level data structure properties.

[0024] The compression module is used to compress and encode the collected data and then transmit it to the acquisition service.

[0025] The parsing module is used by the collection service to parse and process the data, filter out abnormal data, add timestamps, geographic information and personalized attributes, and then forward it to Kafka;

[0026] The streaming processing module is used to consume data from Kafka in real time using streaming services, filter the data by device ID based on the broadcast status, and retain only the data that matches the preset device ID;

[0027] The database writing module is used to write the filtered data into the database;

[0028] The visualization module provides multi-dimensional filtering capabilities through a visual interface, enabling real-time preview and verification of embedded data.

[0029] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to execute the above-described data flow-based embedded point visualization verification method.

[0030] Fourthly, the present invention provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the above-described data flow-based embedded point visualization verification method.

[0031] The main contributions and innovations of this invention are as follows:

[0032] 1. Enable real-time preview and dynamic verification of embedded data.

[0033] By integrating the Flink stream processing engine with Kafka and using a broadcast state JOIN mechanism to process data streams, the data latency is controlled within seconds, solving the problem that the data tracking points cannot be previewed in real time in existing technologies. This allows for real-time verification of the reporting status of interactive behavior tracking points during the development and testing phases.

[0034] 2. Improve data accuracy and data entry reliability

[0035] By configuring device IDs on the front end and combining them with strict matching filtering rules, data from non-target devices is discarded in the streaming service, ensuring that only the data from the specified devices is stored in the database. At the same time, the collection service performs three levels of parsing and verification on the data (gzip decompression → Base64 decoding → format verification), discarding abnormal data and issuing alarms, thus solving the problem of inaccurate data storage in existing technologies.

[0036] 3. Ensure data integrity and traceability

[0037] The visual interface supports filtering by event name and displays all sub-attributes, facilitating manual verification of field integrity; the data collection service adds server time zone timestamps and IP geographic information to avoid client time tampering and data traceability difficulties, solving the defect of existing technologies where embedded data cannot be previewed.

[0038] 4. Optimize system performance and resource utilization.

[0039] The Kafka utility class is initialized using the singleton pattern, which reduces GC pressure by 85% in actual tests and improves system stability. Before data transmission, it is compressed by gzip (compression rate of about 22%) and encoded by Base64 to reduce network transmission load, making it more efficient than existing technologies.

[0040] 5. Reduce the rate of missed detections and false positives in data collection points.

[0041] By using real-time filtering via streaming services and a database write retry mechanism, combined with precise filtering via a visual interface, the false positive rate for 2893 event tracking points was 0 in actual testing, ensuring that no event tracking data was missed and solving the problem of inaccurate verification caused by data loss in existing technologies.

[0042] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0043] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0044] Figure 1 This is a flowchart of a data flow-based visualization verification method for data tracking points according to an embodiment of the present invention;

[0045] Figure 2 This is a flowchart illustrating the compression and encoding process of collected data according to an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram illustrating the acquisition of a device's unique identifier from a front-end page according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram illustrating the addition of a device ID via a front-end page according to an embodiment of the present invention;

[0048] Figure 5 This is a flowchart illustrating the real-time data consumption from Kafka using a streaming service according to an embodiment of the present invention.

[0049] Figure 6 This is a schematic diagram of a visual interface according to an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0052] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0053] Existing technologies have drawbacks such as the inability to preview data collected at tracking points, the inability to preview data in real time, and the inability to accurately store specified data in the database.

[0054] Based on this, the present invention addresses the problems existing in the prior art by using data stream preprocessing (compression, encoding, and secure transmission), dynamic filtering based on Flink broadcast state, near real-time visualization verification, and high availability architecture design.

[0055] Example 1

[0056] This invention aims to propose a data flow-based method for visual verification of event tracking points. Specifically, refer to... Figure 1 The method includes the following steps:

[0057] Step 1: Collect user app interaction data through the event tracking service;

[0058] In this embodiment, the embedded data is defined as shown in Table 1 below:

[0059] Table 1

[0060]

[0061] The following is an example of the raw data for the APP's interactive row data:

[0062] {

[0063] "event": "$AppStart",

[0064] "distinct_id": "5544635",

[0065] "time": 1750082166639,

[0066] "project": "production",

[0067] "$device_id": "5FBA0693-3BA4-4458-91F2-1CF1B9EBE3B0",

[0068] "$manufacturer": "Apple",

[0069] "$os": "iOS",

[0070] "$os_version": "17.4.1",

[0071] "$model": "iPad13,8",

[0072] "$app_version": "8.2.20",

[0073] "$ip": "123.118.208.78",

[0074] "properties": {

[0075] "event_duration": 100

[0076] }

[0077] }

[0078] As can be seen, public attributes are the attribute data that must be carried in all data reporting of event tracking points, and are stored in the first-level data structure; private attributes are personalized attribute data for different events or attribute data added by the server, and are stored in the second-level properties data structure.

[0079] Step 2: Compress and encode the collected data, and then transmit it to the data acquisition service.

[0080] 1. Compress the data using gzip:

[0081] Size before compression: 303 bytes;

[0082] Compressed size: 236 bytes;

[0083] Result: Compressed data was used for transmission.

[0084] 2. Encode the requested data using b64:

[0085] Use Base64.encodeBase64 to convert binary data into ASCII characters (AZ, az, 0-9, +, / ) to avoid errors during transmission and ensure secure data transmission.

[0086] 3. Encode the URL:

[0087] Use URLEncoder.encode to convert special characters into a secure transmission format (% followed by two hexadecimal digits, e.g., space → %20), while decoding is the reverse process (%20 → space). Encoding ensures the reliability of URL transmission, while decoding restores the readability of the original data.

[0088] Final uploaded data (example of data points compressed with gzip and encoded with Base64):

[0089] H4sIAAAAAAAAE+2da3PTxhrHv4uGlwmzu1qtVnlHC8zwqsyh0 / Pi9IxGtpVYg235SHLSDMOMOZCThBASLim3AOW0QFtuBUoJEIbv0kay84qv0Eey44uwHdsxsZ1oJsN4p NVenv3 / 9qbnEf86JThGWhfGsCwhIipMRAqRRwTVsbT4SdVICGOES4rEFZmII4I+qWccYUzQsllV / y6uZx3DzKiWHjethABPjadydlKtz5EykZERIWHYjpGJO0GeAuEiZZI Iz2QtM6tbjqHbwtgpIWfrlqrFjJThTKspKC4FiSkkO2DCfcH46gT8DhJNGlnVdjQnB9ch+4xppbWUWqqYAXVE5XSQZgJqI2ysnS8sPfGuLQmVG9Mp / 4b75EHh6l2 / CDtu 6XpGnTISTlIYo5jCNT / DSd2yoZ2Qlh8kB0UEaWNaQtXicd22q8XZOTurZxJ6AnJwkmrKjJ+E3+NGSq+mSZsJPaU601m / ZG9+wVt965ds2JDOsh01oU0LY+NaytbhapDYb;

[0090] Step 3: The data collection service parses and processes the data, filters out abnormal data, adds timestamps, geographic information, and personalized attributes, and then forwards it to Kafka.

[0091] In this embodiment, as Figure 2 As shown, data parsing is mainly performed through a step-by-step process of URL decoding → Base64 decoding → gzip decompression → public attribute validation, filtering out abnormally formatted data points to ensure that the data structure and content entering downstream (such as Kafka) are compliant and complete.

[0092] in, Figure 2 The logic is as shown in Table 2 below:

[0093] Table 2

[0094]

[0095] In this embodiment, the data processing is as follows:

[0096] 1. Add a timestamp to the server's time zone when data is received, resolving the issue of clients being able to modify the time and preventing incorrect times from being reported.

[0097] 2. Obtain the IP address from the request header, process IPv4 and IPv6 requests through the IP address database, resolve the IP address, and add detailed geographical information such as country and province to the original data.

[0098] 3. Add personalized features, such as user login status detection and platform affiliation.

[0099] The data after adding the attribute is as follows:

[0100] {

[0101] "event": "$AppStart",

[0102] "distinct_id": "5544635",

[0103] "time": 1750082166639,

[0104] "project": "production",

[0105] "$device_id": "5FBA0693-3BA4-4458-91F2-1CF1B9EBE3B0",

[0106] "$manufacturer": "Apple",

[0107] "$os": "iOS",

[0108] "$os_version": "17.4.1",

[0109] "$model": "iPad13,8",

[0110] "$app_version": "8.2.20",

[0111] "$ip": "123.118.208.78",

[0112] "properties": {

[0113] "event_duration": 100,

[0114] "$city": "Beijing",

[0115] "$country": "China",

[0116] "$province": "Beijing",

[0117] "recv_time": 1750082175166,

[0118] }

[0119] }

[0120] In this embodiment, sending data to Kafka is done as follows:

[0121] 1. Initialize the Kafka utility class using the singleton pattern:

[0122] Performance improvement: Avoids frequent object creation and destruction (actual test shows an 85% reduction in GC pressure);

[0123] Memory saving: Only one instance exists in the system memory;

[0124] A globally unique access point: Instances are obtained uniformly through static methods;

[0125] Initialization control: Lazy loading is supported.

[0126] 2. Set the number of retries to ensure successful data transmission.

[0127] Step 4: Consume real-time data from Kafka using streaming services, filter the data by device ID based on broadcast status, and retain only data that matches the preset device ID.

[0128] In this embodiment, device ID filtering is achieved by adding a filtering device:

[0129] 1. For example Figure 3 As shown, by entering the account information on the front-end page, the device's unique identifier, i.e., the device number (ID), can be automatically obtained.

[0130] 2. For example Figure 4 As shown, the device ID is added through the front-end page and automatically sent to the broadcast stream.

[0131] In this embodiment, the filtering rules for device ID filtering are as follows:

[0132] 1. A strict matching mode is adopted, and the device ID in the data points must be consistent with the device ID to be filtered to ensure the validity of the data;

[0133] 2. Filtering operations are performed when the streaming service consumes data from Kafka;

[0134] 3. Data that does not meet the filtering rules will be discarded directly, and only valid data points will be retained.

[0135] Preferably, the streaming service is as follows:

[0136] Objective: To process data in a streaming manner and provide real-time feedback of results;

[0137] Supports open-source engines: Flink / Spark Streaming;

[0138] Using Flink: integrated stream and batch architecture; state management; efficient fault tolerance mechanism; integration with Kafka; broadcast-based dual-stream JOIN; integration with JDBC.

[0139] Preferably, a broadcast-based dual-stream join:

[0140] 1. Broadcast State is a special state type provided by Flink that allows data from one stream to be broadcast to all parallel tasks for association with data from another stream.

[0141] 2. Broadcast stream: Low-throughput dimensional data (device-filtered data stream);

[0142] 3. Main data stream: High-throughput factual data (tracking data stream).

[0143] In this embodiment, JdbcSink is used on the data writing side, and key configurations such as retry and size are set during writing to ensure that the data is successfully entered into the database.

[0144] like Figure 5 As shown, this step mainly utilizes Flink's Broadcast State feature to achieve real-time association between the "high-throughput instrumentation stream" and the "low-throughput device rule stream," solving the problem that traditional "static configuration files cannot dynamically update filtering rules."

[0145] in, Figure 5 The logic is shown in Table 3 below:

[0146] Table 3

[0147]

[0148] As can be seen, this step transmits the filtering rules in real time through Flink broadcast stream, allowing the data from the data points to flow through a sieve like "water". Only the "drops" (target device data) that meet the rules will flow into the database, supporting the core advantage of "precise data entry" of this invention.

[0149] Step 5: Write the filtered data into the database and provide multi-dimensional filtering functions through a visual interface to achieve real-time preview and verification of the data tracking points, such as... Figure 6 As shown.

[0150] In this embodiment, the goal of data visualization is to enhance the visual presentation of filtered data, facilitating data verification and validation by the business side, supporting filtering of devices, and enabling easy viewing of data from specific devices. The specific implementation scheme is as follows:

[0151] The client is as follows:

[0152] 1. Define uniform request parameters:

[0153] {

[0154] "startTime": "",

[0155] "endTime": "",

[0156] "appFrom": "",

[0157] "event": "home_page_uppertab_impression",

[0158] "deviceId": ["AB727BA3-2F71-46F6-86F2-02AEBB383D86"],

[0159] "attrFilter": "",

[0160] "pageNum": 1,

[0161] "pageSize": 10

[0162] }

[0163] 2. Supports page filtering: date, platform, device ID, event, attribute;

[0164] 3. Supports pagination.

[0165] On the server side, it is as follows:

[0166] 1. Define standard structure data return:

[0167] {

[0168] "success": true,

[0169] "code": null,

[0170] "message": null,

[0171] "data": {

[0172] "data": [

[0173] {

[0174] "youthmodel_is_open": false,

[0175] "$model": "iPhone12,1",

[0176] "$os": "iOS",

[0177] "model_type": "Mobile phone",

[0178] "project": "English Fun",

[0179] "user_vip_status": 2,

[0180] "brand_name": "iPhone",

[0181] "$ip": "115.193.78.247",

[0182] "user_style": "student",

[0183] "$app_version": "8.2.20",

[0184] "user_stage": "Office worker",

[0185] "tab_name": "Dubbing",

[0186] "time": "2025-06-19 09:33:36",

[0187] "event": "home_page_uppertab_impression",

[0188] "$is_login_id": true,

[0189] "user_ability_level": "1"

[0190] }

[0191] ],

[0192] "title": {

[0193] "event": "Event Name",

[0194] "project": "Platform Source",

[0195] "time": "time"

[0196] },

[0197] "totalCount": 5

[0198] }

[0199] }

[0200] 2. Dynamically parse the event tracking data, parse the attribute fields, annotate the attribute definitions, and return the data to the front end for processing;

[0201] 3. Use the traditional high-availability architecture of Dubbo + Zookeeper to achieve rapid feature iteration and high availability.

[0202] Example 2

[0203] Based on the same concept, this invention also proposes a data flow-based event tracking visualization verification device, comprising:

[0204] The event tracking module is used to collect user APP interaction data through the event tracking service. The interaction data includes public attributes and private attributes. The public attributes are located in the first-level data structure, and the private attributes are located in the second-level data structure properties.

[0205] The compression module is used to compress and encode the collected data and then transmit it to the acquisition service.

[0206] The parsing module is used by the collection service to parse and process the data, filter out abnormal data, add timestamps, geographic information and personalized attributes, and then forward it to Kafka;

[0207] The streaming processing module is used to consume data from Kafka in real time using streaming services, filter the data by device ID based on the broadcast status, and retain only the data that matches the preset device ID;

[0208] The database writing module is used to write the filtered data into the database;

[0209] The visualization module provides multi-dimensional filtering capabilities through a visual interface, enabling real-time preview and verification of embedded data.

[0210] Example 3

[0211] This embodiment also provides an electronic device, see reference. Figure 7 It includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0212] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0213] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0214] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0215] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the data flow-based embedded point visualization verification methods in the above embodiments.

[0216] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0217] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0218] Input / output device 408 is used to input or output information.

[0219] Example 4

[0220] This embodiment also provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the data flow-based embedded point visualization verification method according to Embodiment 1.

[0221] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0222] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0223] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 1 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0224] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0225] The above embodiments are merely illustrative of several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A data flow-based method for visual verification of event tracking, characterized in that, The method comprises the following steps: Collecting user APP interaction data through a point-in-time collection service, wherein the interaction data comprises public attributes and private attributes, the public attributes are located in a first data structure, and the private attributes are located in a second data structure properties; Compressing the collected data by using gzip, encoding the compressed data by using Base64, and encoding and processing a URL, and transmitting the data to the collection service; The collection service analyzes and processes the data, filters abnormal data, adds a timestamp, geographical information and personalized attributes, and forwards the data to Kafka; Real-time data consumption is performed from Kafka by using a streaming service, device ID filtering is performed on the data based on a broadcast state, a device ID is configured through a front-end interface and is sent to a broadcast stream, a strict matching mode is used to screen the device ID in the point-in-time data, data that does not meet the rules is directly discarded, and only data that matches the preset device ID is reserved; wherein, the Kafka tool class is initialized in a singleton mode, the number of data sending retries is set, and a device filtering rule is configured; the streaming service uses a Flink or SparkStreaming engine, and JOIN processing is performed on a low-throughput device filtering data stream and a high-throughput point-in-time data stream through a broadcast state; The filtered data is written into a database, and a multi-dimensional screening function is provided through a visual interface to realize real-time preview and verification of the point-in-time data. The collection service analyzes and processes the data, which comprises: layer-by-layer decoding and format checking of the encoded data, discarding of non-compliant data and recording of logs, obtaining of a timestamp of a time zone where the server is located and adding of the timestamp to the data, and obtaining of geographical information by resolving an IP address through an IP address library and adding of the geographical information to the data.

2. The data flow based in-app instrumentation visualization verification method of claim 1, wherein, The visual interface supports screening based on time, platform, device ID, event type and attributes, and the returned data comprises structured information of an event name, time, platform source and attribute definition. 3.The data flow based pinpoint visual verification method of claim 1 or 2, wherein, The filtered data is written into a database, which comprises: writing into the database by using a JdbcSink, setting of a writing retry mechanism and batch submission parameters.

4. A data flow based in-app verification of a visualized instrumentation, characterized in that, The method comprises the following steps: The point-in-time collection module is configured to collect user APP interaction data through a point-in-time collection service, wherein the interaction data comprises public attributes and private attributes, the public attributes are located in a first data structure, and the private attributes are located in a second data structure properties; The compression module is configured to compress the collected data by using gzip, encode the compressed data by using Base64, and encode and process a URL, and transmit the data to the collection service; The analysis module is configured to analyze and process the data by the collection service, filter abnormal data, add a timestamp, geographical information and personalized attributes, and forward the data to Kafka; wherein, the collection service analyzes and processes the data, which comprises: layer-by-layer decoding and format checking of the encoded data, discarding of non-compliant data and recording of logs, obtaining of a timestamp of a time zone where the server is located and adding of the timestamp to the data, and obtaining of geographical information by resolving an IP address through an IP address library and adding of the geographical information to the data; The stream processing module is used for real-time consumption of data from Kafka by using a stream service, device ID filtering of the data based on a broadcast state, configuration of the device ID through a front-end interface and delivery to the broadcast stream, screening of the device ID in the trace data by using a strict matching mode, direct discarding of data that does not meet the rules, and retention of only data that matches the preset device ID; wherein, the Kafka tool class is initialized by using a singleton mode, the number of data sending retries is set, and the device filtering rules are configured; the stream service uses a Flink or SparkStreaming engine, and JOIN processing of a low-throughput device filtering data stream and a high-throughput trace data stream is realized by using a broadcast state; The database writing module is used for writing the filtered data into a database. The visual display module is used for providing a multi-dimensional screening function through a visual interface, and realizing real-time preview and verification of the trace data. 5.An electronic device comprising a memory and a processor, the electronic device comprising: The memory stores a computer program, and the processor is configured to run the computer program to execute the data stream based trace visual verification method in any one of claims 1 to 3.

6. A readable storage medium characterized by, The readable storage medium stores a computer program, and the computer program includes program code for controlling a process to execute the process, and the process includes the data stream based trace visual verification method in any one of claims 1 to 3.

Images