Buried point data processing method and system suitable for high-concurrency scene and electronic equipment

Abstract

The invention provides a buried point data processing method suitable for a high-concurrency scene, which is characterized by comprising the following steps of: receiving buried point event data from a client, and writing the buried point event data into a buffer queue of a memory database; through a first timing task configured with a first preset period, acquiring the burying point event data from the buffer queue in batches, and persisting the acquired burying point event data in batches into a local structured text file in an additional writing manner; synchronizing the locally stored structured text file to an object storage service through a second timing task configured with a second preset period; and establishing a mapping relationship with the structured text file in the object storage service through an appearance function of the analytical database. The method has the advantages that the conversion logic and data processing are simple, and the data storage cost is low; the supporting capability of the system on a high-concurrency scene is improved, and meanwhile, the high availability of services is guaranteed; the data loading time is short, and the data link is simplified.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and in particular to a method, system, and electronic device for processing embedded data in high-concurrency scenarios. Background Technology

[0002] The description in this section provides only background information relevant to the disclosure of this application and does not constitute prior art.

[0003] With the rapid development of internet technology, user behavior data analysis has become an important tool for enterprises to optimize products and achieve precise operations. Event tracking technology is the primary method for acquiring user behavior data. Currently, event tracking data analysis solutions in the industry mainly fall into two categories:

[0004] The first category is based on open-source or commercial event tracking systems (such as Matomo and Google Analytics). The disadvantage of this type of solution is that the customizability of event definition and analysis logic is insufficient, and its functions are relatively closed, making it difficult to meet the needs of flexible configuration and complex multi-dimensional analysis of event tracking in enterprise-level scenarios.

[0005] The second category is a self-built big data solution based on log tracking combined with ETL (Extract, Transform, Load) data cleaning processes. While this solution offers high flexibility, it requires building a complete big data platform system, encompassing multiple complex stages such as log collection, message queues, stream processing, data cleaning, and data warehouse storage. This results in high system construction and maintenance costs and a high technical threshold, placing a heavy burden, especially on small and medium-sized enterprises.

[0006] In summary, a significant technical contradiction exists in existing technologies: how to significantly reduce the system's construction complexity and overall cost while ensuring the performance of the data tracking system in handling high-concurrency data and meeting flexible analysis needs. Existing solutions fail to strike a good balance between flexibility, high performance, and low cost.

[0007] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and facilitating understanding by those skilled in the art. It should not be assumed that these technical solutions are known to those skilled in the art simply because they have been described in the background section of this application. Summary of the Invention

[0008] Based on the aforementioned deficiencies in the prior art, this application provides a method, system, and electronic device for processing embedded data in high-concurrency scenarios, which addresses the technical problem that existing embedded data systems cannot simultaneously achieve high concurrency performance, flexible analysis capabilities, and low cost.

[0009] To achieve the above objectives, this application provides the following technical solution: a method for processing event tracking data in high-concurrency scenarios, comprising the following steps:

[0010] Receive event data from the client and write the event data into the buffer queue of the in-memory database;

[0011] By configuring a first timed task with a first preset period, the event data is retrieved in batches from the buffer queue, and the retrieved batch of event data is persisted to a local structured text file by appending it.

[0012] The structured text file stored locally is synchronized to the object storage service by a second timed task configured with a second preset period.

[0013] By leveraging the external attributes of the analytical database, a mapping relationship is established between the database and the structured text files in the object storage service.

[0014] Preferably, the step "receiving event data from the client and writing the event data into the buffer queue of the in-memory database" includes:

[0015] The received event data is standardized and its validity is verified.

[0016] Preferably, the step "by configuring a first timed task with a first preset period to retrieve the event data in batches from the buffer queue, and persisting the retrieved batch of event data to a local structured text file by appending it" includes:

[0017] After confirming that the event tracking data has been successfully persisted to the structured text file, the event tracking data is removed from the buffer queue.

[0018] Preferably, the structured text file is a CSV format file.

[0019] Preferably, the path naming of the structured text file includes a version identifier and time partition information.

[0020] Preferably, the first preset period is in the second range, and the second preset period is in the hour range.

[0021] This application discloses a data processing system for embedded data points, including:

[0022] The data collection layer includes a data collection API interface and a data buffer queue. The data collection API interface is used to receive data collection event data from the client. The data buffer queue includes a memory database with a buffer layer. The data buffer queue is used to receive data collection event data from the data collection API interface in protocol buffer format and write it to the buffer layer.

[0023] The data persistence layer is used to retrieve the event data from the buffer queue in batches through a first timed task configured with a first preset period, and persist the retrieved event data to a local structured text file by appending it.

[0024] The data storage and data warehouse integration layer is used to synchronize the locally stored structured text files to the object storage service through a second scheduled task configured with a second preset period, and then establish a mapping relationship with the structured text files in the object storage service through the external table function of the analytical database.

[0025] Preferably, the data persistence layer is further used for:

[0026] After confirming that the data has been successfully persisted to the structured text file, the data buffer module is notified to remove the data from the buffer queue.

[0027] This application discloses an electronic device, including:

[0028] One or more processors;

[0029] Memory, used to store one or more programs.

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for processing embedded data in high-concurrency scenarios.

[0031] This application discloses a computer-readable storage medium storing a computer program thereon, characterized in that the program, when executed by a processor, implements the aforementioned method for processing embedded data in high-concurrency scenarios.

[0032] The beneficial effects of this application, based on the above technical solutions, are as follows:

[0033] This invention systematically addresses the shortcomings of existing technologies through the following technological innovations:

[0034] 1. Data storage layer optimization:

[0035] By storing the data points in CSV format and using its structured column mode to clearly define the data, the complex transformation logic in the traditional ETL process can be eliminated, reducing the complexity of data processing.

[0036] Compared to JSON format, CSV uses omitting key fields for storage, reducing storage overhead by approximately 30%-50% and significantly lowering data storage costs.

[0037] 2. High-concurrency processing technology:

[0038] By introducing Redis's Push / Pop queue mechanism to build a buffer layer, asynchronous writing of event tracking is achieved, improving the system's ability to support high-concurrency scenarios (theoretically, the peak processing capacity can reach 100,000 times / second), while ensuring high availability of the service.

[0039] 3. Improved data warehouse integration efficiency:

[0040] Leveraging the OSS CSV parsing engine built into Alibaba Cloud AnalyticDB, one-click import of CSV files into the data warehouse is achieved through native external table association technology, reducing data loading time by 70% compared to traditional ETL processes and simplifying the data chain.

[0041] Specific embodiments of this application are disclosed in detail with reference to the following description and accompanying drawings, illustrating how the principles of this application can be employed. It should be understood that the embodiments of this application are not limited in scope. Within the spirit and scope of the appended claims, the embodiments of this application include many changes, modifications, and equivalents.

[0042] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0043] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, whole, step, or component, but does not exclude the presence or addition of one or more other features, wholes, steps, or components. Attached Figure Description

[0044] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this application in any way. Furthermore, the shapes and scales of the components in the drawings are merely illustrative to aid in understanding this application and do not specifically limit the shapes and scales of the components. Those skilled in the art, guided by the teachings of this application, can select various possible shapes and scales to implement this application according to specific circumstances. In the drawings:

[0045] Figure 1This is a flowchart of a method for processing embedded data according to an embodiment of the present invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0047] It should be noted that when an element is referred to as being "set on" another element, it can be directly on the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only embodiments.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0049] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0050] It should be noted that in the description of this application, the terms "first," "second," etc., are used only for descriptive purposes and to distinguish similar objects; there is no order between them, nor should they be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more.

[0051] Reference Figure 1 As shown, this invention provides a method for processing embedded data based on a layered architecture. The specific process is as follows: Embedded data acquisition layer

[0052] - Tracking API Interface: Designed with a standardized RESTful interface to receive tracking event data sent by clients (such as Web and App clients). The interface protocol supports HTTP / HTTPS transmission and integrates a data validation mechanism to ensure the legality of input.

[0053] - Data Buffer Queue: A distributed queue is built through the Redis cluster. The API interface pushes the received event data to the Redis List data structure in real time in the format of Protocol Buffers, forming a high-concurrency write buffer layer.

[0054] Data persistence layer

[0055] - Scheduled data persistence task (first scheduled task): Deploy a background service that uses a timer to pull data from the Redis queue in batches every N seconds (configurable) (Pop operation), and persist the data to a local CSV file using an append write mode.

[0056] - File naming conventions: CSV file paths follow hierarchical naming rules:

[0057] {version} / date={yyyyMMdd} / hour={HH} / {csvFilename}.csv

[0058] Here, {version} is the version number in CSV format, and {yyyyMMdd} and {HH} correspond to the data collection date and time period, respectively, enabling efficient indexing of data by version and time dimension.

[0059] Data storage and data warehouse integration layer

[0060] -OSS File Synchronization Task (Second Scheduled Task): Triggers a batch upload operation of local CSV files to Alibaba Cloud OSS once per hour through the scheduled service, ensuring that scheduled uploads do not cause congestion.

[0061] - One-click data warehouse import: Utilizing AnalyticDB's ExternalTable feature, a predefined CSV table structure bound to the OSS path is used to import data via standard SQL statements.

[0062]

[0063]

[0064] This process automatically triggers AnalyticDB's built-in CSV parser to perform data type mapping and quality verification, and directly loads the data into the target table of the data warehouse.

[0065] The autonomous analysis system can retrieve data from the target table in the data warehouse to query, analyze, or process the acquired event tracking data.

[0066] This invention claims protection for a data tracking architecture design based on existing components. Through innovative combination of technical elements and process optimization, it forms a technical solution that differs from traditional solutions. The main advantages are as follows:

[0067] 1. High-concurrency asynchronous buffer-disk persistence architecture

[0068] -Technology combination: Utilize Redis distributed queue (List) to receive event tracking (Push) in real time, and use configurable scheduled tasks (every N seconds) to pull (Pop) in batches and write to disk in CSV format.

[0069] -Innovation: Decoupling acquisition and storage, improving concurrent processing capabilities through "queue buffering + batch writing" and avoiding the performance bottleneck of traditional synchronous writing.

[0070] 2. Low-cost data warehouse integration chain

[0071] - Technology combination: Build a three-tiered pipeline of "local CSV → OSS → AnalyticDB" using scheduled tasks:

[0072] - Synchronize CSV to OSS every hour;

[0073] -Utilize AnalyticDB's external table functionality to directly parse OSS files and import them into the data warehouse with a single click, eliminating the need for ETL.

[0074] -Innovation: By skipping the traditional data cleaning process, AnalyticDB's native capabilities enable a seamless "storage-analysis" connection, reducing the cost and complexity of the process.

[0075] 3. Structured data definition and version management

[0076] - Technical features: The data format for data tracking is defined using a CSV column mode, and multiple versions of data can be stored in parallel through versioned file naming rules (such as {version} / date=xxx / ...).

[0077] -Innovation: Leveraging the structured nature of CSV to reduce metadata dependency, and achieving rapid data retrieval and traceability through hierarchical filename design.

[0078] 4. Multi-dimensional scheduled task collaboration mechanism

[0079] -Technical features: Design an N-second (data persistence to disk) and hourly (OSS synchronization / data warehouse import) scheduled task matrix, and manage timing dependencies and fault tolerance logic through a scheduling system.

[0080] -Innovation: Differentiated configuration of task cycles balances real-time performance and resource consumption, ensuring the efficiency and stability of data processing across the entire chain.

[0081] ETL stands for Extract, Transform, and Load, a combined process used to standardize the processing of heterogeneous data sources into a data warehouse.

[0082] OSS stands for Alibaba Cloud Object Storage Service, which provides highly reliable, low-cost massive data storage capabilities.

[0083] AnalyticDB is a distributed analytical database developed by Alibaba Cloud, which supports real-time data import and interactive analysis.

[0084] This application discloses a data processing system for embedded data points, including:

[0085] The data collection layer includes a data collection API interface and a data buffer queue. The data collection API interface is used to receive data collection event data from the client. The data buffer queue includes a memory database with a buffer layer. The data buffer queue is used to receive data collection event data from the data collection API interface in protocol buffer format and write it to the buffer layer.

[0086] The data persistence layer is used to retrieve the event data from the buffer queue in batches through a first timed task configured with a first preset period, and persist the retrieved event data to a local structured text file by appending it.

[0087] The data storage and data warehouse integration layer is used to synchronize the locally stored structured text files to the object storage service through a second scheduled task configured with a second preset period, and then establish a mapping relationship with the structured text files in the object storage service through the external table function of the analytical database.

[0088] The event tracking API interface can adopt a standardized RESTful interface to receive event tracking data sent by clients (such as Web or App clients). The interface protocol supports HTTP / HTTPS transmission and integrates a data validation mechanism to ensure the legality of the input. Of course, in other optional implementations, the event tracking API interface can also adopt other standardized interfaces, non-standardized interfaces, or other interface protocols to obtain data, and use other optional mechanisms to ensure the legality of the input.

[0089] Data buffer queues can be built using a Redis cluster to create a distributed queue. The API interface pushes received event data in real time to a Redis List data structure in Protocol Buffers format, forming a high-concurrency write buffer layer. Of course, in other alternative implementations, data buffer queues can be built using other database formats.

[0090] The data persistence layer can deploy background services via scheduled data write-to-disk tasks. A timer is used to batch retrieve data from the Redis queue every N seconds (configurable) using a pop operation, and the data is persisted to a local CSV file using an append-only write mode. Of course, in other optional implementations, the structured text file can also be other forms of local file. When naming the CSV file, its file path follows a hierarchical naming convention: {version} / date={yyyyMMdd} / hour={HH} / {csvFilename}.csv

[0091] Here, {version} represents the version number in CSV format, and {yyyyMMdd} and {HH} correspond to the data collection date and time period, respectively, enabling efficient indexing of data by version and time dimension. Of course, in other optional methods, the file path can also be designed according to other naming rules.

[0092] The data storage and data warehouse integration layer is used to trigger a batch upload operation of local CSV files to Alibaba Cloud OSS once per hour via a scheduled service. Scheduled uploads avoid congestion. It utilizes AnalyticDB's ExternalTable feature to predefine CSV external table structures bound to OSS paths, and executes data import using standard SQL statements. Of course, other alternative implementations can also use other object storage services, not limited to Alibaba Cloud OSS; other analytical databases, not limited to AnalyticDB, can also be used.

[0093] This application also discloses an electronic device, including:

[0094] One or more processors;

[0095] Memory, used to store one or more programs.

[0096] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method for processing data tracking points in high-concurrency scenarios.

[0097] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for processing embedded data in high-concurrency scenarios.

[0098] It should be understood that the above description is for illustrative purposes and not for limitation. Many embodiments and applications beyond the provided examples will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this teaching should not be determined by reference to the above description, but rather by reference to the foregoing claims and the full scope of their equivalents. For purposes of completeness, all articles and references, including patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the foregoing claims is not intended as a waiver of that subject matter, nor should it be construed as an indication that the applicant has not considered that subject matter as part of the disclosed application subject matter.

Claims

1. A method for processing event tracking data in high-concurrency scenarios, characterized in that, Includes the following steps: Receive event data from the client and write the event data into the buffer queue of the in-memory database; By configuring a first timed task with a first preset period, the event data is retrieved in batches from the buffer queue, and the retrieved batch of event data is persisted to a local structured text file by appending it. The structured text file stored locally is synchronized to the object storage service by a second timed task configured with a second preset period. By leveraging the external attributes of the analytical database, a mapping relationship is established between the database and the structured text files in the object storage service.

2. The method according to claim 1, characterized in that, The step "receiving event data from the client and writing the event data into the buffer queue of the in-memory database" includes: The received event data is standardized and its validity is verified.

3. The method for processing embedded data in high-concurrency scenarios according to claim 1, characterized in that, The step "by configuring a first timed task with a first preset period to retrieve the event data in batches from the buffer queue, and persisting the retrieved batch of event data to a local structured text file by appending it" includes: After confirming that the event tracking data has been successfully persisted to the structured text file, the event tracking data is removed from the buffer queue.

4. The method for processing embedded data in high-concurrency scenarios according to claim 1, characterized in that, The structured text file is a CSV format file.

5. The method for processing embedded data in high-concurrency scenarios according to claim 1, characterized in that, The path naming of the structured text file includes a version identifier and time partition information.

6. The method for processing embedded data in high-concurrency scenarios according to claim 1, characterized in that, The first preset period is in seconds, and the second preset period is in hours.

7. A data processing system for embedded data points, characterized in that, include: The data collection layer includes a data collection API interface and a data buffer queue. The data collection API interface is used to receive data collection event data from the client. The data buffer queue includes a memory database with a buffer layer. The data buffer queue is used to receive data collection event data from the data collection API interface in protocol buffer format and write it to the buffer layer. The data persistence layer is used to retrieve the event data from the buffer queue in batches through a first timed task configured with a first preset period, and persist the retrieved event data to a local structured text file by appending it. The data storage and data warehouse integration layer is used to synchronize the locally stored structured text files to the object storage service through a second scheduled task configured with a second preset period, and then establish a mapping relationship with the structured text files in the object storage service through the external table function of the analytical database.

8. The system according to claim 7, characterized in that, The data persistence layer is also used for: After confirming that the data has been successfully persisted to the structured text file, the data buffer module is notified to remove the data from the buffer queue.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method for high-concurrency scenarios as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the data processing method for high-concurrency scenarios as described in any one of claims 1 to 6.

Images