Data processing method, device and equipment and readable storage medium
By introducing cache processing channels, delay queues and processing chains in buried point reporting data processing, the problem of inefficient data processing under large concurrency is solved, and near-real-time statistical and persistent processing is realized.
Patent Information
- Application Number
- CN202411996660.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-18
AI Technical Summary
The existing buried-point reporting data processing methods are inefficient under large concurrency, resulting in statistical calculations and persistence dragging down the entire reporting data processing process.
By receiving the reported data from the buried point, the collected data corresponding to the statistical indicators are stored in the cache processing channel. Consumers regularly obtain data from the cache processing channel, and after depositing different indicator caches according to the statistical indicator type, notification data is sent to the corresponding delay queue. The collected data is preprocessed and statistically processed using the delay queue and processing chain.
Improve the efficiency of reporting data processing, realize near-real-time statistics of various applications and complete persistence, without the need to introduce additional message middleware.
Smart Images
Figure CN119847895A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of information processing technology, and in particular to a data processing method, device, equipment and readable storage medium. Background Art
[0002] Tracking analysis is a common data collection method for website analysis. After receiving the reported data from tracking, it is necessary to count some information about the visit, such as PV (Page View, page views or clicks), UV (UniqueVisitor, unique visitor number), IP (Internet Protocol, unique IP refers to the total number of IPs that have visited a site), etc.
[0003] In the general tracking point reporting method, directly processing these reported data will not meet the needs of large concurrency, because statistical calculation and persistence will drag down the entire reporting data processing process. Some indicators need to be deduplicated, so in general processing solutions, message queues are used to process them. Adding message queue middleware will increase resources and make the entire reporting process take too long.
[0004] In summary, how to effectively solve the problems such as the reported data processing efficiency is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention
[0005] The purpose of this application is to provide a data processing method, device, equipment and readable storage medium to improve the efficiency of reporting data processing, thereby achieving near real-time statistics of indicators of various applications and completing persistence.
[0006] In order to solve the above technical problems, this application provides the following technical solutions:
[0007] A data processing method, comprising:
[0008] Receive the buried point report data, and store the collected data corresponding to the statistical indicators in the buried point report data into the cache processing channel;
[0009] Using consumers to periodically obtain the collected data from the cache processing channel, and after storing the collected data in different indicator caches according to statistical indicator types, sending notification data to corresponding delay queues;
[0010] Using the delay queue and combining with the notification data, respectively pre-processing the collected data in different indicator buffers, and storing the processed target collected data into the processing chain;
[0011] The target collected data is processed using the processing chain to obtain statistical results corresponding to the statistical indicators.
[0012] Preferably, storing the collected data corresponding to the statistical indicators in the tracking point reporting data into a cache processing channel includes:
[0013] Encapsulating the buried point reporting data into an entity;
[0014] The entity is stored in the cache processing channel.
[0015] Preferably, storing the entity into the cache processing channel comprises:
[0016] Based on the load balancing strategy, determining a target partition from a plurality of partitions in the cache processing channel;
[0017] The entity is stored in the queue of the target partition.
[0018] Preferably, the consumer is used to periodically obtain the collected data from the cache processing channel, and after storing the collected data in different indicator caches according to the statistical indicator type, the notification data is sent to the corresponding delay queue, including:
[0019] Using the consumer to periodically obtain the entity from the cache processing channel having a mapping relationship;
[0020] Decapsulating the entity to obtain the collected data;
[0021] According to the correspondence between the statistical indicator type and the indicator buffer, the collected data is stored in different indicator buffers;
[0022] Send notification data to the corresponding delay queue.
[0023] Preferably, the delayed queue is used in combination with the notification data to perform corresponding preprocessing on the collected data in different indicator buffers, including:
[0024] After a specified delay, taking out the notification data using the delay queue;
[0025] Determining a statistical indicator type from a notification type of the notification data;
[0026] The processors corresponding to the different statistical indicator types are called to perform corresponding preprocessing on the collected data in the corresponding indicator cache.
[0027] Preferably, the collected data in the corresponding indicator buffer are preprocessed accordingly, including:
[0028] Expired data in the collected data is filtered out.
[0029] Preferably, the target acquisition data is processed by the processing chain to obtain statistical results corresponding to the statistical indicators, including:
[0030] If the target collection data corresponds to the page views, superimposing the target collection data and the persisted page views to obtain the latest page views;
[0031] After obtaining the latest page views, persist the latest page views and delete the target collected data;
[0032] If the target collected data corresponds to the number of independent visitors, deduplication processing is performed on the target collected data, and the number of independent visitors is determined using the deduplication-processed target collected data;
[0033] If the target collection data corresponds to the number of independent IPs, the target collection data is deduplicated according to the statistical time, and the number of independent IPs is determined using the deduplicated target collection data.
[0034] A data processing device, comprising:
[0035] A data cache module is used to receive the buried point report data and store the collected data corresponding to the statistical indicators in the buried point report data into a cache processing channel;
[0036] A data distribution module, used for periodically acquiring the collected data from the cache processing channel by using consumers, and after storing the collected data into different indicator caches according to statistical indicator types, sending notification data to corresponding delay queues;
[0037] A data preprocessing module, used to use the delay queue and the notification data to perform corresponding preprocessing on the collected data in different indicator buffers, and store the processed target collected data into a processing chain;
[0038] The data statistics module is used to process the target collected data using the processing chain to obtain statistical results corresponding to the statistical indicators.
[0039] An electronic device, comprising:
[0040] Memory for storing computer programs;
[0041] A processor is used to implement the steps of the above data processing method when executing the computer program.
[0042] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above data processing method are implemented.
[0043] Apply the method provided in the embodiment of the present application to receive tracking point reporting data, and store the collected data corresponding to the statistical indicators in the tracking point reporting data into a cache processing channel; use consumers to periodically obtain collected data from the cache processing channel, and after storing the collected data into different indicator caches according to the statistical indicator type, send notification data to the corresponding delay queue; use the delay queue and combine with the notification data to perform corresponding preprocessing on the collected data in different indicator caches, and store the target collected data obtained by processing into a processing chain; use the processing chain to process the target collected data to obtain statistical results corresponding to the statistical indicators.
[0044] In the present application, after receiving the buried point report data, the collected data corresponding to the statistical indicators in the buried point report data are first stored in the buffer processing channel. That is, a large amount of data corresponding to the buried point report data is screened, and only the collected data corresponding to the statistical indicators are stored in the cache processing channel. Then, the consumer can regularly obtain the collected data from the cache processing channel, and after storing the collected data in different indicator caches according to the statistical indicator type, send notification data to the corresponding delay queue. In other words, the consumer can divert the collected data so that it can be stored in different indicator caches. Diverting the collected data and storing it in different indicator caches can further facilitate accelerated processing. The delay queue can convert the collected data into batch processing instead of processing them one by one. Specifically, the delay queue is used in combination with the notification data to perform corresponding preprocessing on the collected data in different indicator caches, and the target collected data obtained by processing is stored in the processing chain. Finally, the target collected data is processed using the processing chain to obtain the statistical results corresponding to the statistical indicators. In other words, this application does not need to introduce additional message middleware. With the help of cache and delay queues, it can effectively accelerate the processing of point-of-care reporting data, improve the processing efficiency of point-of-care reporting data, and thus achieve near real-time statistics of indicators of each application and complete persistence.
[0045] Correspondingly, the embodiments of the present application also provide data processing devices, equipment and readable storage media corresponding to the above-mentioned data processing method, which have the above-mentioned technical effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 This is a flowchart of a data processing method in an embodiment of the present application;
[0048] Figure 2 A schematic diagram of a data processing method in an embodiment of the present application;
[0049] Figure 3 This is a schematic diagram of the structure of a data processing device in an embodiment of the present application;
[0050] Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application;
[0051] Figure 5 This is a schematic diagram of the specific structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0052] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0053] For ease of description, the relevant technical terms are explained in detail below:
[0054] PV (Page View): Page views or clicks, that is, the total number of times all pages on a website are visited. Each time a page is opened or refreshed, PV increases once, and the number of views accumulates when the same page is opened or refreshed multiple times. PV reflects the number of web pages visited by website users and is one of the important indicators for measuring website traffic.
[0055] UV (Unique Visitor): The number of unique visitors refers to the number of unique users who visit a website. Multiple visits by the same visitor in one day are counted only once. It can be understood as the number of computers that visit a website. The website determines the identity of the visiting computer through the cookies of the visiting computer. If you change the IP but do not clear the cookies and visit the same website again, the number of UVs in the statistics of the website will remain unchanged. Among them, cookies are data fragment files created when visiting a website. By saving browsing information, users can have a better online experience.
[0056] IP (Internet Protocol): Independent IP refers to the total number of IPs that have visited a site, with the user's IP address as the statistical basis. The same IP address is only counted once between 00:00 and 24:00. The IP address is the identity of the device in the network. By counting the number of IP addresses, the number of independent devices visiting the website can be estimated.
[0057] Please refer to Figure 1 , Figure 1 This is a flow chart of a data processing method in an embodiment of the present application, the method comprising the following steps:
[0058] S101, receiving the buried point reporting data, and storing the collected data corresponding to the statistical indicators in the buried point reporting data into a cache processing channel.
[0059] In this embodiment, for application software that needs to perform indicator statistical analysis, tracking points can be set up, and data reporting is performed when user behavior triggers tracking point reporting conditions (such as the number of collected data reaches a preset threshold) or reporting methods such as scheduled reporting. The reported data includes but is not limited to user ID, IP address, application information, and reporting date.
[0060] The tracking point reporting data refers to the data reported after being collected by the tracking points on the application software to be used for indicator statistical analysis.
[0061] Since there are many types of data reported by the tracking point, and some of the data is irrelevant to the statistical indicators, after receiving the tracking point report data, only the collected data corresponding to the statistical indicators in the tracking point report data can be stored in the buffer processing channel. The data information corresponding to the indicator that needs to be counted is collected. For example, if only IP is counted, the collected data is IP; if only UV is counted, the collected data is ID; if only PV is counted, the collected data is the reported application and date. If only IP, UV and PV are counted, the collected data is ID, IP, reported application and date.
[0062] In a specific implementation of the present application, the collected data corresponding to the statistical indicators in the tracking point reporting data is stored in the cache processing channel, including:
[0063] Encapsulate the tracking point reporting data into entities;
[0064] Stores the entity into the cache processing channel.
[0065] The step of storing the entity into a cache processing channel includes:
[0066] Based on the load balancing strategy, determine the target partition from multiple partitions in the cache processing channel;
[0067] Store the entity in the queue of the target partition.
[0068] For ease of description, the above steps are combined for explanation below.
[0069] Among them, the composition of the data in the encapsulated entity can be specified by the statistical indicators required by the user: if the statistical indicator is only UV, the user ID in the tracking point reporting data can be encapsulated; if the statistical indicator is only IP, the IP in the tracking point reporting data needs to be decapsulated; if it is only the statistical indicator PV, then only the reporting application and date in the tracking point reporting data need to be decapsulated. Of course, if the statistical indicators include multiple, the corresponding entity encapsulated data composition will correspond to multiple. For example, if the statistical indicators include UV, PV and IP, the data encapsulated by the entity includes IP, ID, reporting application and date in the tracking point reporting data.
[0070] Please refer to Figure 2 In this embodiment, the cache processor can be divided into multiple channels, each channel corresponds to a consumer. Each channel is divided into different partitions, and each partition can be set as an unbounded queue or an array queue according to user needs to divert data. When each encapsulated entity data enters, a partition can be selected through load balancing to be put in, and the specified consumer will obtain data from these partitions within a period of time.
[0071] S102: using consumers to periodically obtain collected data from a cache processing channel, and after storing the collected data in different indicator caches according to statistical indicator types, sending notification data to corresponding delay queues.
[0072] The cache processor is divided into multiple channels, each channel has a separate consumer, each consumer can have its own consumer logic for different processing flows; if the processing flows are the same, then only one consumer is needed.
[0073] In this embodiment, the consumer can periodically obtain the collected data from the corresponding buffer processing channel, and store the collected data in different indicator buffers according to the statistical indicator type, and send a notification message to the response delay queue.
[0074] In this embodiment, consumers are used to periodically obtain collected data from the cache processing channel, and after storing the collected data in different indicator caches according to statistical indicator types, notification data is sent to the corresponding delay queue, including:
[0075] Use consumers to periodically obtain entities from the cache processing channel with mapping relationships;
[0076] Decapsulate the entity to obtain the collected data;
[0077] According to the correspondence between statistical indicator types and indicator buffers, the collected data is stored in different indicator buffers;
[0078] Send notification data to the corresponding delay queue.
[0079] For ease of description, the above steps are combined for explanation below.
[0080] Consumers can periodically obtain entities from the cache channel for decapsulation and consumption, and store the data in different indicator caches. In this embodiment, consumers can simply divide the collected data corresponding to the statistical indicators in the buried data by date, application, and indicator, and store them in different indicator caches. After completing the indicator diversion, notify the corresponding delay queue.
[0081] S103 , using the delay queue and in combination with the notification data, respectively pre-processing the collected data in different indicator buffers, and storing the processed target collected data into the processing chain.
[0082] The delay queue can determine that the corresponding collected data has been stored in different indicator buffers based on the notification message. Then, the collected data in different indicator buffers can be preprocessed separately. In practical applications, different preprocessing methods can be used for the collected data in different indicator buffers.
[0083] After the preprocessing is completed, the processed target acquisition data is stored in the processing chain.
[0084] In a specific implementation of the present application, the collected data in different indicator buffers are preprocessed respectively by using a delay queue and combining notification data, including:
[0085] After the specified delay, the notification data is retrieved using the delay queue;
[0086] Determine the statistical indicator type from the notification type of the notification data;
[0087] The processors corresponding to different statistical indicator types are called to pre-process the collected data in the corresponding indicator cache.
[0088] The collected data in the corresponding indicator buffer are preprocessed accordingly, including:
[0089] Filter out expired data from collected data.
[0090] For ease of description, the above steps are combined for explanation below.
[0091] The delay queue can retrieve the notification data after the specified delay and obtain the corresponding processor from the notification type of the notification data. For example, for IP type, the processor that processes the IP is determined, and for UV type, the processor that processes the application and date is determined. The processor will process some dirty data, such as filtering expired data, and then obtain target collection data such as application and date, and put the target collection data into the processing chain.
[0092] S104: Use the processing chain to process the target collected data to obtain statistical results corresponding to the statistical indicators.
[0093] The processing chain is a chain structure formed by organizing various data processing operations in a specific form, so that these operations are executed continuously in a fixed order. In this embodiment, the processing chain corresponds to target collection data of different statistical indicators and can correspond to different processing methods, thereby obtaining statistical results corresponding to different statistical indicators.
[0094] The processing chain processes the target collected data to obtain the indicator statistics based on the embedded data. In actual applications, the processing chain processes the target collected data and can lock the cache. The lock here is mainly to ensure that critical data can be accessed concurrently.
[0095] In a specific implementation of the present application, the target acquisition data is processed using a processing chain to obtain statistical results corresponding to the statistical indicators, including:
[0096] If the target collected data corresponds to the page views, the target collected data and the persisted page views are superimposed to obtain the latest page views;
[0097] After obtaining the latest page views, persist the latest page views and delete the target collected data;
[0098] If the target collected data corresponds to the number of unique visitors, deduplication processing is performed on the target collected data, and the number of unique visitors is determined using the deduplication-processed target collected data;
[0099] If the target collection data corresponds to the number of independent IPs, the target collection data is deduplicated according to the statistical time, and the number of independent IPs is determined using the deduplicated target collection data.
[0100] For ease of description, the above steps are combined for explanation below.
[0101] The processing chain handles different indicators in this way. If it is PV (processing strategy: multiple accumulation), then first get the key from the entity. The key here can be composed of application and date. Then get all the corresponding data from the pv cache, calculate the sum of these data, add them to the persistent data, and then delete them. If there are some indicators that need to be deduplicated, such as UV (unchanged), IP (unchanged for static IP or new IP covers old IP for dynamic IP), these data do not need to be deleted, because these data need to be deduplicated and need to be deleted after the indicators on this date are invalid.
[0102] Based on the statistical results, the growth of visits to all applications can be displayed, so as to know which application functions are actively used by users and which application functions are not used.
[0103] Apply the method provided in the embodiment of the present application to receive tracking point reporting data, and store the collected data corresponding to the statistical indicators in the tracking point reporting data into a cache processing channel; use consumers to periodically obtain collected data from the cache processing channel, and after storing the collected data into different indicator caches according to the statistical indicator type, send notification data to the corresponding delay queue; use the delay queue and combine with the notification data to perform corresponding preprocessing on the collected data in different indicator caches, and store the target collected data obtained by processing into a processing chain; use the processing chain to process the target collected data to obtain statistical results corresponding to the statistical indicators.
[0104] In the present application, after receiving the buried point report data, the collected data corresponding to the statistical indicators in the buried point report data are first stored in the buffer processing channel. That is, a large amount of data corresponding to the buried point report data is screened, and only the collected data corresponding to the statistical indicators are stored in the cache processing channel. Then, the consumer can regularly obtain the collected data from the cache processing channel, and after storing the collected data in different indicator caches according to the statistical indicator type, send notification data to the corresponding delay queue. In other words, the consumer can divert the collected data so that it can be stored in different indicator caches. Diverting the collected data and storing it in different indicator caches can further facilitate accelerated processing. The delay queue can convert the collected data into batch processing instead of processing them one by one. Specifically, the delay queue is used in combination with the notification data to perform corresponding preprocessing on the collected data in different indicator caches, and the target collected data obtained by processing is stored in the processing chain. Finally, the target collected data is processed using the processing chain to obtain the statistical results corresponding to the statistical indicators. In other words, this application does not need to introduce additional message middleware. With the help of cache and delay queues, it can effectively accelerate the processing of point-of-care reporting data, improve the processing efficiency of point-of-care reporting data, and thus achieve near real-time statistics of indicators of each application and complete persistence.
[0105] Corresponding to the above method embodiment, the embodiment of the present application further provides a data processing device. The data processing device described below and the data processing method described above can refer to each other.
[0106] See also Figure 3 As shown, the device includes the following modules:
[0107] The data cache module 101 is used to receive the buried point report data and store the collected data corresponding to the statistical indicators in the buried point report data into the cache processing channel;
[0108] The data distribution module 102 is used to obtain the collected data from the cache processing channel regularly by using consumers, and after storing the collected data into different indicator caches according to the statistical indicator type, send notification data to the corresponding delay queue;
[0109] The data preprocessing module 103 is used to perform corresponding preprocessing on the collected data in different indicator buffers by using the delay queue and combining the notification data, and store the processed target collected data into the processing chain;
[0110] The data statistics module 104 is used to process the target collected data using the processing chain to obtain statistical results corresponding to the statistical indicators.
[0111] The device provided in the embodiment of the present application is used to receive the tracking point reporting data, and store the collected data corresponding to the statistical indicators in the tracking point reporting data into a cache processing channel; use consumers to periodically obtain the collected data from the cache processing channel, and after storing the collected data into different indicator caches according to the statistical indicator type, send notification data to the corresponding delay queue; use the delay queue and combine with the notification data to perform corresponding pre-processing on the collected data in different indicator caches, and store the target collected data obtained by processing into a processing chain; use the processing chain to process the target collected data to obtain statistical results corresponding to the statistical indicators.
[0112] In the present application, after receiving the buried point report data, the collected data corresponding to the statistical indicators in the buried point report data are first stored in the buffer processing channel. That is, a large amount of data corresponding to the buried point report data is screened, and only the collected data corresponding to the statistical indicators are stored in the cache processing channel. Then, the consumer can regularly obtain the collected data from the cache processing channel, and after storing the collected data in different indicator caches according to the statistical indicator type, send notification data to the corresponding delay queue. In other words, the consumer can divert the collected data so that it can be stored in different indicator caches. Diverting the collected data and storing it in different indicator caches can further facilitate accelerated processing. The delay queue can convert the collected data into batch processing instead of processing them one by one. Specifically, the delay queue is used in combination with the notification data to perform corresponding preprocessing on the collected data in different indicator caches, and the target collected data obtained by processing is stored in the processing chain. Finally, the target collected data is processed using the processing chain to obtain the statistical results corresponding to the statistical indicators. In other words, this application does not need to introduce additional message middleware. With the help of cache and delay queues, it can effectively accelerate the processing of point-of-care reporting data, improve the efficiency of point-of-care reporting data processing, and thus achieve near real-time statistics of indicators of each application and complete persistence.
[0113] In a specific implementation of the present application, the data cache module is specifically used to encapsulate the buried point reporting data into an entity;
[0114] Stores the entity into the cache processing channel.
[0115] In a specific implementation of the present application, the data cache module is specifically used to determine a target partition from multiple partitions in a cache processing channel based on a load balancing strategy;
[0116] Store the entity in the queue of the target partition.
[0117] In a specific implementation of the present application, the data distribution module is specifically used to use consumers to periodically obtain entities from a cache processing channel having a mapping relationship;
[0118] Decapsulate the entity to obtain the collected data;
[0119] According to the correspondence between statistical indicator types and indicator buffers, the collected data is stored in different indicator buffers;
[0120] Send notification data to the corresponding delay queue.
[0121] In a specific implementation of the present application, the data preprocessing module is specifically used to retrieve the notification data using the delay queue after a specified delay;
[0122] Determine the statistical indicator type from the notification type of the notification data;
[0123] The processors corresponding to different statistical indicator types are called to perform corresponding preprocessing on the collected data in the corresponding indicator cache.
[0124] In a specific implementation of the present application, the data preprocessing module is specifically used to filter out expired data in the collected data.
[0125] In a specific implementation of the present application, the data statistics module is used to superimpose the target collected data and the persisted page views to obtain the latest page views if the target collected data corresponds to the page views;
[0126] After obtaining the latest page views, persist the latest page views and delete the target collected data;
[0127] If the target collected data corresponds to the number of unique visitors, deduplication processing is performed on the target collected data, and the number of unique visitors is determined using the deduplication-processed target collected data;
[0128] If the target collection data corresponds to the number of independent IPs, the target collection data is deduplicated according to the statistical time, and the number of independent IPs is determined using the deduplicated target collection data.
[0129] Corresponding to the above method embodiment, an embodiment of the present application further provides an electronic device. The electronic device described below and the data processing method described above can refer to each other.
[0130] See also Figure 4 As shown, the electronic device includes:
[0131] A memory 332, for storing computer programs;
[0132] The processor 322 is used to implement the steps of the data processing method of the above method embodiment when executing a computer program.
[0133] For details, please refer to Figure 5 , Figure 5 A schematic diagram of the specific structure of an electronic device provided for this embodiment, which may have relatively large differences due to different configurations or performances, may include one or more processors (central processing units, CPU) (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the electronic device 301.
[0134] The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .
[0135] The steps in the data processing method described above can be implemented by the structure of an electronic device.
[0136] Corresponding to the above method embodiment, the embodiment of the present application further provides a readable storage medium. The readable storage medium described below and the data processing method described above can refer to each other.
[0137] A readable storage medium stores a computer program, which implements the steps of the data processing method of the above method embodiment when the computer program is executed by a processor.
[0138] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.
[0139] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0140] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0141] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0142] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms include, include or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0143] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data processing method, characterized in that: include: Receive the buried point report data, and store the collected data corresponding to the statistical indicators in the buried point report data into the cache processing channel; Using consumers to periodically obtain the collected data from the cache processing channel, and after storing the collected data in different indicator caches according to statistical indicator types, sending notification data to corresponding delay queues; Using the delay queue and combining with the notification data, respectively pre-processing the collected data in different indicator buffers, and storing the processed target collected data into the processing chain; The target collected data is processed using the processing chain to obtain statistical results corresponding to the statistical indicators.
2. The method according to claim 1, characterized in that Storing the collected data corresponding to the statistical indicators in the tracking point reporting data into a cache processing channel, including: Encapsulating the buried point reporting data into an entity; The entity is stored in the cache processing channel.
3. The method according to claim 2, characterized in that Storing the entity into the cache processing channel includes: Based on the load balancing strategy, determining a target partition from a plurality of partitions in the cache processing channel; The entity is stored in the queue of the target partition.
4. The method according to claim 2, characterized in that: Using consumers to periodically obtain the collected data from the cache processing channel, and after storing the collected data in different indicator caches according to statistical indicator types, sending notification data to the corresponding delay queue, including: Using the consumer to periodically obtain the entity from the cache processing channel having a mapping relationship; Decapsulating the entity to obtain the collected data; According to the correspondence between the statistical indicator type and the indicator buffer, the collected data is stored in different indicator buffers; Send notification data to the corresponding delay queue.
5. The method according to claim 1, characterized in that Using the delay queue and combining the notification data, the collected data in different indicator buffers are preprocessed accordingly, including: After a specified delay, taking out the notification data using the delay queue; Determining a statistical indicator type from a notification type of the notification data; The processors corresponding to the different statistical indicator types are called to perform corresponding preprocessing on the collected data in the corresponding indicator cache.
6. The method according to claim 5, characterized in that The collected data in the corresponding indicator buffer are preprocessed accordingly, including: Expired data in the collected data is filtered out.
7. The method according to any one of claims 1 to 6, characterized in that: The target collected data is processed by using the processing chain to obtain statistical results corresponding to the statistical indicators, including: If the target collection data corresponds to the page views, superimposing the target collection data and the persisted page views to obtain the latest page views; After obtaining the latest page views, persist the latest page views and delete the target collected data; If the target collected data corresponds to the number of independent visitors, deduplication processing is performed on the target collected data, and the number of independent visitors is determined using the deduplication-processed target collected data; If the target collection data corresponds to the number of independent IPs, the target collection data is deduplicated according to the statistical time, and the number of independent IPs is determined using the deduplicated target collection data.
8. A data processing device, characterized in that: include: A data cache module is used to receive the buried point report data and store the collected data corresponding to the statistical indicators in the buried point report data into a cache processing channel; A data distribution module, used for periodically acquiring the collected data from the cache processing channel by using consumers, and after storing the collected data into different indicator caches according to statistical indicator types, sending notification data to corresponding delay queues; A data preprocessing module, used to use the delay queue and the notification data to perform corresponding preprocessing on the collected data in different indicator buffers respectively, and store the processed target collected data into a processing chain; The data statistics module is used to process the target collected data using the processing chain to obtain statistical results corresponding to the statistical indicators.
9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program.
10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.