Unlock instant, AI-driven research and patent intelligence for your innovation.

Data collection method, device and terminal equipment

A data acquisition and data technology, applied in the field of data processing, can solve the problems of the amount of data exceeding the maximum temporarily stored data, the reduction of real-time performance of cumulative data, and the reduction of accuracy of cumulative data, etc., to achieve small data volume and avoid data errors , Guaranteed real-time effect

Active Publication Date: 2022-02-18
BEIJING YIZHEN XUESI EDUCATION TECH CO LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] When an exception occurs in the spark cluster, the offset when reading the Kafka message queue is usually adjusted to re-read the failed data in the Kafka message queue. However, if the spark cluster is down for a long time, it will The data that caused the read failure to exceed the maximum amount of temporarily stored data in the Kafka message queue, resulting in a reduction in data accuracy, which in turn leads to a reduction in the accuracy of the determined cumulative data
When the Kafka message queue is abnormal, it will read the data of the backup Kafka message queue from the upstream database, and make statistics based on the read data to determine the cumulative data. However, because the backup data is not It will be updated in real time according to the real-time data in the Kafka message queue, resulting in a reduction in the real-time performance of the read data, which in turn will result in a reduction in the real-time performance of the determined cumulative data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data collection method, device and terminal equipment
  • Data collection method, device and terminal equipment
  • Data collection method, device and terminal equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0025] refer to figure 1 , showing a flow chart of the steps of a data acquisition method according to Embodiment 1 of the present invention.

[0026] The data collection method of this embodiment includes the following steps:

[0027] S101. From at least two preset candidate storage areas, determine a first storage area for storing real-time data of the user in the current time period.

[0028] In this embodiment, the preset at least two candidate storage areas may be storage areas in the database, or may be storage areas in other locations, as long as real-time data can be stored, which is not limited in this embodiment.

[0029] In this embodiment, the current time period may be the time period corresponding to the current time point, for example, one minute, one hour or one day where the current time point is located may be used as the current time period.

[0030] Specifically, the current time period may be the time period within the current time period, and the time p...

Embodiment 2

[0049] refer to Figure 2a , showing a flow chart of the steps of a data collection method according to Embodiment 2 of the present invention.

[0050] The data collection method of this embodiment includes the following steps:

[0051] S201. From the preset at least two candidate storage areas, determine a first storage area for storing real-time data of the user's current time period according to a preset switching rule.

[0052] In this embodiment, the switching rule may be a switching rule for defining a candidate storage area as the first storage area. Those skilled in the art may set specific switching rules as required, which is not limited in this embodiment.

[0053] In an implementation manner of this embodiment, the preset switching rule includes: the at least two candidate storage areas are periodically switched sequentially according to a preset switching sequence to serve as the first storage area.

[0054] Specifically, assuming that the number of candidate s...

Embodiment 3

[0097] refer to image 3 , showing a flow chart of the steps of a data collection method according to Embodiment 3 of the present invention.

[0098] This embodiment uses Kafka message queue, Spark cluster, and Redis database as examples to describe the data collection process of user user_1.

[0099] The data collection method in this embodiment includes the following steps, all of which can be implemented by a Spark cluster:

[0100] S301. From the preset two candidate storage areas user_1_A or user_1_B, determine a first storage area for storing real-time data after switching according to a preset switching rule.

[0101] Among them, user_1_A and user_1_B are the indexes of the two alternative storage areas respectively, which may point to the storage areas in the Redis database.

[0102] Specifically, the index corresponding to the first storage area may be specifically indicated by the first indication mark as user_1_A or user_1_B.

[0103] The alternative storage area...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

Embodiments of the present invention provide a data collection method, device and terminal equipment. The data collection method includes: from the preset at least two candidate storage areas, determine the first storage area for storing the real-time data of the user's current time period; collect the user data in the message queue in real time to obtain the real-time data, and The real-time data is accumulated and stored in the first storage area; according to the timing acquisition rules, the timing data corresponding to the real-time data accumulated and stored in the first time period is generated, and the timing data is accumulated and stored in the preset third storage area, wherein the first The real-time data of the time period is stored in the second storage area, the second storage area is an alternative storage area in at least two alternative storage areas except the first storage area, and the first time period is the time before the current time period part. The accumulative data is determined through the data stored in the scheme provided by this embodiment, which can ensure the accuracy and real-time performance of the accumulative data.

Description

technical field [0001] Embodiments of the present invention relate to the technical field of data processing, and in particular, to a data collection method, apparatus, and terminal device. Background technique [0002] In the process of collecting data, it is often necessary to collect cumulative data, such as the number of logins, the number of browsing, and the like. [0003] The existing process of collecting cumulative data can be as follows: first use the spark cluster to read the data in the Kafka message queue, and then perform statistics according to the read data to determine the cumulative data; When an exception occurs or the Kafka message queue fails to read data from the Kafka message queue, other alternative methods will be used to read data. [0004] When an exception occurs in the spark cluster, the offset when reading the Kafka message queue is usually adjusted to re-read the failed data in the Kafka message queue. However, if the spark cluster goes down f...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/958G06F9/54G06F16/27
CPCG06F16/958G06F9/547G06F16/27G06F2209/541
Inventor 贾延夺温宗霖王亮
Owner BEIJING YIZHEN XUESI EDUCATION TECH CO LTD