A data sampling method, device and computer readable storage medium

Abstract

The present application discloses a data sampling method, device and computer-readable storage medium, the method comprising: sampling data of a target type at a first sampling rate in a first time window to obtain target sampling data; merging the target sampling data with historical sampling data in a sampling database to obtain merged sampling data, wherein a third sampling rate of the merged sampling data is determined according to the first sampling rate and a second sampling rate of the historical sampling data, when the amount of the merged sampling data is greater than a sampling amount threshold, sampling the merged sampling data based on an adaptive sampling parameter to obtain updated historical sampling data, the sampling amount threshold being related to the data of the target type; the adaptive sampling parameter is used to control the amount of the merged sampling data within the sampling amount threshold; and replacing the historical sampling data in the sampling database with updated historical sampling data. By adopting the present application, system capacity can be saved when sampling data.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a data sampling method, device and computer-readable storage medium. Background Art

[0002] With the continuous development of computer networks, the amount of data generated in computer networks is increasing day by day. When it is necessary to perform big data statistical analysis on the full amount of data in the computer network, first of all, it is necessary to perform sampling statistics on the full amount of data, and then the data obtained by the sampling statistics can be used to characterize the full amount of data. For example, the data features of the data obtained by the sampling statistics can be used as the data features of the full amount of data.

[0003] In the prior art, when it is necessary to sample data generated within a certain time period, it is usually necessary to read all the data within the time period in advance, and then use a set sampling rate to sample the data within the time period. Since it is necessary to read all the data within the time period in advance in the prior art, when the amount of data within the time period is too large, a high requirement is placed on the system capacity. Summary of the invention

[0004] The present application provides a data sampling method, device, and computer-readable storage medium, which can save system capacity when sampling data.

[0005] On one hand, the present application provides a data sampling method, comprising:

[0006] Sampling the data of the target type at a first sampling rate in a first time window to obtain target sampled data;

[0007] Merging the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data, wherein a third sampling rate of the merged sampling data is determined according to the first sampling rate and the second sampling rate of the historical sampling data, the historical sampling data is sampling data of the target type obtained by sampling data of the target type before obtaining the target sampling data, the second sampling rate is the sampling rate of the historical sampling data relative to the data of the target type, and the third sampling rate is the sampling rate of the merged sampling data relative to the data of the target type;

[0008] When the amount of combined sampling data is greater than the sampling amount threshold, the combined sampling data is sampled based on the adaptive sampling parameter to obtain updated historical sampling data, the sampling amount threshold is related to the data of the target type, and the adaptive sampling parameter is used to control the amount of combined sampling data within the sampling amount threshold;

[0009] Replace the historical sampling data in the sampling database with the updated historical sampling data.

[0010] The method further includes:

[0011] A fourth sampling rate is determined based on the adaptive sampling parameters and the third sampling rate so that the target type of data is sampled using the fourth sampling rate as the first sampling rate in a second time window, wherein the fourth sampling rate is a sampling rate for updating historical sampling data relative to the target type of data, and the second time window is the next time window of the first time window.

[0012] The target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including:

[0013] When the first sampling rate is greater than the second sampling rate, determining a ratio between the second sampling rate and the first sampling rate as a first ratio sampling rate;

[0014] Sampling the target sampling data according to the first ratio sampling rate to obtain sampled target sampling data;

[0015] The target sampling data and the historical sampling data after sampling are determined as the merged sampling data.

[0016] The target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including:

[0017] When the first sampling rate is less than the second sampling rate, determining a ratio between the first sampling rate and the second sampling rate as a second ratio sampling rate;

[0018] Sampling the historical sampling data according to the second ratio sampling rate to obtain sampled historical sampling data;

[0019] Determine the historical sampling data and target sampling data after sampling as merged sampling data;

[0020] When the first sampling rate is equal to the second sampling rate, the target sampling data and the historical sampling data are determined as the merged sampling data.

[0021] The method of sampling the target type of data at a first sampling rate in a first time window to obtain target sampled data includes:

[0022] Determine the second sampling rate of the historical sampling data as the first sampling rate;

[0023] In a first time window, data of a target type is sampled using a first sampling rate to obtain target sampled data;

[0024] Then, the target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including:

[0025] The target sampling data and the historical sampling data are determined as merged sampling data.

[0026] When the amount of combined sampling data is greater than the sampling amount threshold, the combined sampling data is sampled based on the adaptive sampling parameter to obtain updated historical sampling data, including:

[0027] When the amount of combined sampling data is greater than the sampling amount threshold, obtaining a data identification string corresponding to the combined sampling data;

[0028] Map the data identification string to the uniform sampling space to obtain the hash value corresponding to the data identification string;

[0029] The combined sampling data is sampled according to the third sampling rate, the adaptive sampling parameter and the hash value to obtain updated historical sampling data.

[0030] The combined sampling data is sampled according to the third sampling rate, the adaptive sampling parameter and the hash value to obtain updated historical sampling data, including:

[0031] Obtaining a fourth sampling rate according to the third sampling rate and the adaptive sampling parameter;

[0032] Based on the fourth sampling rate and the hash value, the merged sampling data is sampled to obtain updated historical sampling data; the fourth sampling rate is the sampling rate of the updated historical sampling data relative to the data of the target type.

[0033] Among them, it also includes:

[0034] When the amount of the combined sampling data is less than or equal to the sampling amount threshold, the historical sampling data in the sampling database is replaced with the combined sampling data.

[0035] The previous time window of the first time window is the third time window; the first time window and the third time window have an intersection time window; the historical sampling data is the sampling data obtained by sampling the target type of data in the intersection time window of the third time window;

[0036] Sampling data of a target type at a first sampling rate in a first time window to obtain target sampled data includes:

[0037] In a time window other than the intersection time window of the first time window, sampling the data of the target type at a first sampling rate to obtain target sampled data;

[0038] Then, the target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including:

[0039] When the number of historical sampling data is less than the combined number threshold, and the ratio between the first sampling rate and the second sampling rate is less than the ratio threshold, the historical sampling data is deleted, and the target sampling data is determined as the combined sampling data.

[0040] The method of sampling the target type of data at a first sampling rate in a first time window to obtain target sampled data includes:

[0041] Reading data of the target type from the data stream in the first time window, performing field parsing on the read data of the target type to obtain initial parsed data;

[0042] Filter multiple field information in the initial parsed data based on the filtering mechanism to obtain filtered parsed data;

[0043] Based on the vocabulary association mechanism, associated field information is added to the filtered and parsed data to obtain sampled business data;

[0044] The sampled service data is sampled at a first sampling rate to obtain target sampled data.

[0045] The method of sampling the target type of data at a first sampling rate in a first time window to obtain target sampled data includes:

[0046] In a first time window, sampling data of a target type at a first sampling rate through a first thread to obtain target sampled data;

[0047] Then, the target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including:

[0048] The target sampling data and the historical sampling data in the sampling database are merged through the second thread to obtain merged sampling data.

[0049] On one hand, the present application provides a data sampling device, comprising:

[0050] A sampling module, used for sampling data of a target type at a first sampling rate in a first time window to obtain target sampled data;

[0051] a merging module, configured to merge the target sampling data and the historical sampling data in the sampling database to obtain the merged sampling data, wherein the third sampling rate of the merged sampling data is determined according to the first sampling rate and the second sampling rate of the historical sampling data, the historical sampling data is the sampling data of the target type obtained by sampling the data of the target type before obtaining the target sampling data, the second sampling rate is the sampling rate of the historical sampling data relative to the data of the target type, and the third sampling rate is the sampling rate of the merged sampling data relative to the data of the target type;

[0052] An adaptive sampling module, used for sampling the merged sampling data based on an adaptive sampling parameter to obtain updated historical sampling data when the amount of merged sampling data is greater than a sampling amount threshold, wherein the sampling amount threshold is related to the data of the target type, and the adaptive sampling parameter is used to control the amount of merged sampling data within the sampling amount threshold;

[0053] The replacement module is used to replace the historical sampling data in the sampling database with updated historical sampling data.

[0054] Wherein, the data sampling device is also used for:

[0055] A fourth sampling rate is determined based on the adaptive sampling parameters and the third sampling rate so that the target type of data is sampled using the fourth sampling rate as the first sampling rate in a second time window, wherein the fourth sampling rate is a sampling rate for updating historical sampling data relative to the target type of data, and the second time window is the next time window of the first time window.

[0056] Among them, the merge module includes:

[0057] A first ratio determination unit, configured to determine a ratio between the second sampling rate and the first sampling rate as a first ratio sampling rate when the first sampling rate is greater than the second sampling rate;

[0058] A first ratio sampling unit, used for sampling the target sampling data according to a first ratio sampling rate to obtain the sampled target sampling data;

[0059] The first merging unit is used to determine the sampled target sampling data and the historical sampling data as merged sampling data.

[0060] Among them, the merge module includes:

[0061] A second ratio determination unit, configured to determine a ratio between the first sampling rate and the second sampling rate as a second ratio sampling rate when the first sampling rate is less than the second sampling rate;

[0062] A second ratio sampling unit, used for sampling the historical sampling data according to a second ratio sampling rate to obtain the sampled historical sampling data;

[0063] A second merging unit, used for determining the sampled historical sampling data and the target sampling data as merged sampling data;

[0064] The third merging unit is used to determine the target sampling data and the historical sampling data as merged sampling data when the first sampling rate is equal to the second sampling rate.

[0065] The sampling module includes:

[0066] A sampling rate determination unit, used to determine the second sampling rate of the historical sampling data as the first sampling rate;

[0067] A window sampling unit, used to sample data of a target type at a first sampling rate within a first time window to obtain target sampled data;

[0068] Then, the merge module is also used for:

[0069] The target sampling data and the historical sampling data are determined as merged sampling data.

[0070] Among them, the adaptive sampling module includes:

[0071] A character string acquisition unit, used for acquiring a data identification character string corresponding to the combined sampling data when the amount of combined sampling data is greater than a sampling amount threshold;

[0072] A mapping unit, used to map the data identification string into a uniform sampling space to obtain a hash value corresponding to the data identification string;

[0073] The combined sampling unit is used to sample the combined sampling data according to the third sampling rate, the adaptive sampling parameter and the hash value to obtain updated historical sampling data.

[0074] The combined sampling unit includes:

[0075] An adaptive sampling rate subunit, configured to obtain a fourth sampling rate according to the third sampling rate and the adaptive sampling parameter;

[0076] The merged sampling subunit is used to sample the merged sampling data based on a fourth sampling rate and a hash value to obtain updated historical sampling data; the fourth sampling rate is a sampling rate of the updated historical sampling data relative to the target type of data.

[0077] The data sampling device is also specifically used for:

[0078] When the amount of the combined sampling data is less than or equal to the sampling amount threshold, the historical sampling data in the sampling database is replaced with the combined sampling data.

[0079] The previous time window of the first time window is the third time window; the first time window and the third time window have an intersection time window; the historical sampling data is the sampling data obtained by sampling the target type of data in the intersection time window of the third time window;

[0080] The sampling module is also used for:

[0081] In a time window other than the intersection time window of the first time window, data of the target type is sampled at a first sampling rate to obtain target sampled data.

[0082] Then, the merge module is also used for:

[0083] When the number of historical sampling data is less than the combined number threshold, and the ratio between the first sampling rate and the second sampling rate is less than the ratio threshold, the historical sampling data is deleted, and the target sampling data is determined as the combined sampling data.

[0084] Among them, the sampling module includes:

[0085] A data reading unit, used to read data of a target type from a data stream in a first time window, and perform field parsing on the read data of the target type to obtain initial parsed data;

[0086] A filtering unit, used to filter multiple field information in the initial parsed data based on a filtering mechanism to obtain filtered parsed data;

[0087] An association unit, used to add associated field information to the filtered and parsed data based on a vocabulary association mechanism to obtain sampled business data;

[0088] The target sampling unit is used to sample the sampling service data at a first sampling rate to obtain target sampling data.

[0089] Among them, the sampling module is also used for:

[0090] In a first time window, sampling data of a target type at a first sampling rate through a first thread to obtain target sampled data;

[0091] Then, the merge module is also used for:

[0092] The target sampling data and the historical sampling data in the sampling database are merged through the second thread to obtain merged sampling data.

[0093] In one aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes a method in one aspect of the present application.

[0094] In one aspect, the present application provides a computer-readable storage medium storing a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the method in the above aspect.

[0095] The present application can sample the data of the target type at a first sampling rate in a first time window to obtain the target sampling data; merge the target sampling data and the historical sampling data in the sampling database to obtain the merged sampling data, wherein the third sampling rate of the merged sampling data is determined according to the first sampling rate and the second sampling rate of the historical sampling data, and the historical sampling data is the sampling data of the target type obtained by sampling the data of the target type before obtaining the target sampling data; when the number of the merged sampling data is greater than the sampling number threshold, the merged sampling data is sampled based on the adaptive sampling parameter to obtain the updated historical sampling data, the sampling number threshold is related to the data of the target type, and the adaptive sampling parameter is used to control the number of the merged sampling data within the sampling number threshold; replace the historical sampling data in the sampling database with the updated historical sampling data. It can be seen that the method proposed in the present application can further sample the sampling data when the number of the merged sampling data is greater than the sampling number threshold. In this way, during the sampling period or after the sampling is completed, the number of the sampling data stored in the sampling database can always be controlled within the sampling number threshold range, thereby saving system capacity and realizing adaptive adjustment of the sampling rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0097] Figure 1 It is a schematic diagram of a system architecture provided by this application;

[0098] Figure 2a This is a schematic diagram of a data sampling scenario provided by this application;

[0099] Figure 2b This is a schematic diagram of a thread sampling scenario provided by this application;

[0100] Figure 3 It is a flowchart of a data sampling method provided by this application;

[0101] Figure 4 It is a schematic diagram of a time window scenario provided by this application;

[0102] Figure 5 This is a schematic diagram of a scenario for obtaining merged sampling data provided by the present application;

[0103] Figure 6This is a flowchart of obtaining and updating historical sampling data provided by this application;

[0104] Figure 7 This is another flowchart for obtaining and updating historical sampling data provided by the present application;

[0105] Figure 8 is a schematic diagram of another data sampling scenario provided by this application;

[0106] Fig. 9 It is a structural schematic diagram of a data sampling device provided by the present application;

[0107] Fig.10 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0108] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0109] See also Figure 1 , is a schematic diagram of a system architecture provided by this application. Figure 1 As shown, the system architecture diagram includes a device cluster 400a, a server 100 and a device cluster 400b. Among them, the device cluster 400a may include multiple terminal devices, specifically including terminal devices 200a, terminal devices 200b and terminal devices 200c. Among them, the terminal device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile Internet device (mobile internet device, MID), a wearable device (such as a smart watch, a smart bracelet, etc.). The device cluster 400b may include multiple sampling devices, and the sampling device can be a server. Therefore, the multiple sampling devices specifically include a server 300a, a server 300b and a server 300c. Among them, the terminal device in the above-mentioned device cluster 400a can generate data for sampling, and the server 100 can store the data for sampling generated by the terminal device in the device cluster 400a. The sampling device in the device cluster 400b can obtain the data to be sampled from the server 100 and sample the obtained data.

[0110] Please also see Figure 2a , is a schematic diagram of a data sampling scenario provided by this application. Figure 2a As shown, the device cluster 106a includes multiple terminal devices (the device cluster 106a is equivalent to the above Figure 1 In the device cluster 400a in the device cluster 106a, the multiple terminal devices can all be used to respond to the user's operation instructions and generate corresponding data. For example, the terminal device can respond to the user's operation instruction to place an order for a certain product and generate corresponding order information. The order information is the corresponding data generated by the terminal device according to the user's operation instruction. The data generated by the terminal device can not only be a certain type of data (such as order type data), but also include multiple types of data. For example, it can also include data of types such as click records of a certain product being clicked by a user. Therefore, it can be understood that a large amount of data can be obtained through the multiple terminal devices in the above-mentioned device cluster 106a, and the large amount of data can include multiple types of data. The data generated by the above-mentioned terminal device cluster 106a can be the data in the data set 100a. The data set 100a includes multiple data, specifically including n data. The value of n is determined according to the actual application scenario and is not limited to this. For example, n can be 10,000 or 100 million. Among them, one piece of data in the data set 100a can refer to the one piece of order information generated above, or can refer to a click record of a certain product being clicked by a user. In the above Figure 1 The data set 100a is stored in the server 100 in the embodiment.

[0111] The present application mainly provides a method for data sampling, wherein the sampled data may be data generated by multiple terminal devices in the above-mentioned device cluster 106a. In other words, the method provided by the present application is described here by taking the sampling of n data in the data set 100a as an example. Multiple threads can be used to read n data in the data set 100a at the same time, and sample the n data in the data set 100a. It can be understood that the multiple threads can be threads in the sampling device, and multiple sampling devices can be provided to read and sample the data in the data set 100a. Among them, the sampling device can be a server, and a sampling device can also include multiple threads. Therefore, it is necessary to merge the sampling results obtained by multiple threads in all sampling devices in the end. Specifically, first, for a sampling device, it is necessary to merge the sampling results obtained by its multiple threads to obtain the sampling results corresponding to the sampling device. Then, it is necessary to merge the sampling results corresponding to each sampling device (each sampling device can be understood as a thread of a higher level), and finally obtain the sampling results corresponding to all the data in the data set 100a. Here, three threads (such as Figure 2a As shown in FIG. 1 , a thread s1, a thread s2, and a thread s3 are used to read and sample the data in the data set 100a. The three threads may be three different threads in the same sampling device or threads in different sampling devices. For example, the three threads may be Figure 1 In fact, more threads (which may be threads in multiple sampling devices) can be used to read and sample the data in the data set 100a. The number of threads and the number of sampling devices are determined according to the actual application scenario and are not limited thereto.

[0112] Among them, each of thread s1, thread s2 and thread s3 includes multiple data pools, and different data pools are used to store different types of data sampled by the threads. Specifically, thread s1 includes data pool a1, data pool b1 and data pool c1; thread s2 includes data pool a2, data pool b2 and data pool c2; thread s3 includes data pool a3, data pool b3 and data pool c3. It should be noted that data pool a1, data pool a2 and data pool a3 are used to store the same type of data sampled from data set 100a. Data pool b1, data pool b2 and data pool b3 are used to store the same type of data sampled from data set 100a. Data pool c1, data pool c2 and data pool c3 are used to store the same type of data sampled from data set 100a. Assume that the data pools a1, a2 and a3 are used to store the first type of data, the data pools b1, b2 and b3 are used to store the second type of data, and the data pools c1, c2 and c3 are used to store the third type of data. Threads s1, s2 and s3 can read data in the data set 100a at the same time, and sample the read data during the process of reading the data, that is, sample while reading.

[0113] It should be noted that the data in the above data set 100a can be a data stream, that is, in the process of threads 1, 2 and 3 reading the data in the data set 100a and sampling the read data, the data set 100a can continuously increase the data generated by the device cluster 106a, that is, the sampling method provided in the present application can be real-time sampling. In other words, the sampled data in the present application can be data generated in real time and read and sampled by multiple threads in real time. Among them, each piece of data in the data set 100a can include multiple field information, and each thread (including thread s1, thread s2 and thread s3) can identify and classify the read data through the multiple field information corresponding to each piece of data. For example, each thread can classify the read data into income type data, age type data, click type data or order type data, etc. Here, it is assumed that there are three types of data, namely the first type of data (such as income type data), the second type of data (such as age type data) and the third type of data (such as order type data). Thread 1 can sample the first type of data obtained by reading, and the sampling result is the first type of data finally retained in data pool a1, assuming that it is sampling result A1. Thread 1 can sample the second type of data obtained by reading, and the sampling result is the second type of data finally retained in data pool b1, assuming that it is sampling result B1. Thread 1 can sample the third type of data obtained by reading, and the sampling result is the third type of data finally retained in data pool c1, assuming that it is sampling result C1. Thread 2 can sample the first type of data obtained by reading, and the sampling result is the first type of data finally retained in data pool a2, assuming that it is sampling result A2. Thread 2 can sample the second type of data obtained by reading, and the sampling result is the second type of data finally retained in data pool b2, assuming that it is sampling result B2. Thread 2 can sample the third type of data obtained by reading, and the sampling result is the third type of data finally retained in data pool c2, assuming that it is sampling result C2. Thread 3 can sample the first type of data obtained by reading, and the sampling result is the first type of data finally retained in data pool a3, assuming that it is sampling result A3. Thread 3 can sample the second type of data obtained by reading, and the sampling result is the second type of data finally retained in data pool b3, assuming that it is sampling result B3. Thread 3 can sample the third type of data obtained by reading, and the sampling result is the third type of data finally retained in data pool c3, assuming that it is sampling result C3. Therefore, it can be understood that each type of data corresponds to a sampling result, and the sampling processes between different types of data are independent of each other. The sampling result for the above-mentioned first type of data is the result obtained by merging the above-mentioned sampling results A1, sampling results A2 and sampling results A3, such as. Figure 2a The sampling result 101a in the above second type of data is the result obtained by merging the above sampling result B1, sampling result B2 and sampling result B3, such as Figure 2a The sampling result 102a in the above third type of data is the result obtained by merging the above sampling result C1, sampling result C2 and sampling result C3, such as Figure 2a The sampling result 101a corresponding to the first type of data, the sampling result 102a of the second type of data and the sampling result 103a of the third type of data may be collectively referred to as a sampling result 104a obtained by sampling the data in the data set 100a.

[0114] The sampling result 104a includes the above-mentioned sampling result 101a, the sampling result 102a and the above-mentioned sampling result 103a. Figure 2a As shown, in the sampling result 104a, the sampling result 101a specifically samples 3 pieces of the first type of data (respectively, the first type of data 1, the first type of data 2, and the first type of data 3), and the 3 pieces of the first type of data are sampled at a sampling rate of 1 / 24 (i.e., the probability of each piece of data being sampled and retained is 1 / 24). The sampling result 102a specifically samples 4 pieces of the second type of data (respectively, the second type of data 1, the second type of data 2, the second type of data 3, and the second type of data 4), and the 4 pieces of the second type of data are sampled at a sampling rate of 1 / 8 (i.e., the probability of each piece of data being sampled and retained is 1 / 8). The sampling result 103a specifically samples 5 pieces of the third type of data (respectively, the third type of data 1, the third type of data 2, the third type of data 3, the third type of data 4, and the third type of data 5), ​​and the 5 pieces of the third type of data are sampled at a sampling rate of 1 / 4 (i.e., the probability of each piece of data being sampled and retained is 1 / 4). The sampling rate corresponding to each type of data will be adaptively adjusted and changed according to the corresponding sampling quantity threshold during the sampling process. The adjustment process of the sampling rate can be seen in the following Figure 2bDescription in the corresponding embodiment. It should be noted that in actual application scenarios, the above-mentioned sampling results 101a, sampling results 102a and sampling results 103a may include more data obtained by sampling (for example, 100,000 data, 1 million data, etc.), and only 3, 4 and 5 data are used as examples for explanation here. The data features of the 3 first-type data contained in the sampling result 101a obtained above can be used to characterize the data features of all the first-type data in the data set 100a; the data features of the 4 second-type data contained in the sampling result 102a obtained above can be used to characterize the data features of all the second-type data in the data set 100a; the data features of the 5 third-type data contained in the sampling result 103a obtained above can be used to characterize the data features of all the third-type data in the data set 100a.

[0115] For example, assuming that the first type of data is data on the income of each citizen, when it is necessary to count the per capita income of citizens in each region (here, assuming that it includes region 1, region 2, and region 3), the first type of data in the sampling result 101a can be statistically calculated (i.e., the per capita income of citizens in each region is calculated), and the per capita income corresponding to region 1 is 3,000 yuan, the per capita income corresponding to region 2 is 6,000 yuan, and the per capita income corresponding to region 3 is 8,000 yuan. The process of performing statistical calculations on the first type of data in the sampling result 101a can be performed by another device dedicated to statistical calculations, or by the sampling device where any of the threads s1, s2, and s3 are located. The device that performs statistical calculations is determined according to the actual application scenario and is not limited thereto. After performing statistical calculations on the first type of data in the sampling result 101a and obtaining the calculation result (i.e., the per capita income corresponding to region 1 is 3,000 yuan, the per capita income corresponding to region 2 is 6,000 yuan, and the per capita income corresponding to region 3 is 8,000 yuan), the device that obtains the calculation result can send the calculation result to the user terminal 105a, and the user terminal 105a can display the calculation result in the terminal page. The user terminal 105a can be a terminal held by an administrator or a terminal containing an administrator account. The administrator can be an authenticated user with data management authority. For example, the administrator can be a senior manager of a government agency who can obtain the income status of citizens. Figure 2a As shown, the calculation result displayed in the user terminal 105a is "the statistical analysis result of income distribution: Region 1: per capita income of 3,000 yuan; Region 2: per capita income of 6,000 yuan; Region 3: per capita income of 8,000 yuan".

[0116] Since the total amount of each type of data collected is usually extremely large in actual application scenarios, each type of data can be sampled first in the above manner. When it is necessary to perform statistical calculations on the data features contained in each type of data in the data set 100a later, the data features contained in each type of data obtained by sampling can be directly calculated. In this way, since the number of data involved in the calculation is reduced, the amount of calculation for calculating the data features of each type of data can be reduced, and the computational complexity of calculating the data features of each type of data can be reduced. In addition, since the sampling process for each type of data ensures that each piece of data is sampled with equal probability, the data features of the sampled data can be used to characterize the data features of all data.

[0117] The method provided by the present application can set a sampling quantity threshold for each type of sampled data, and the sampling quantity threshold can be understood as the capacity of each data pool. Therefore, each of the above data pools can sample the read data according to the corresponding sampling quantity threshold, so that the number of data finally sampled in each data pool is not greater than the corresponding sampling quantity threshold. Specifically, the sampling quantity thresholds corresponding to different data pools in multiple threads corresponding to a type of sampled data are the same. For example, the sampling quantity threshold corresponding to the above-mentioned first type of data is set to 50, indicating that the number of first type of data that needs to be sampled is less than or equal to 50, then the sampling quantity thresholds corresponding to the above-mentioned data pool a1, data pool a2 and data pool a3 are all 50. Afterwards, when the sampling result A1 corresponding to data pool a1, the sampling result A2 corresponding to data pool a2 and the sampling result A3 corresponding to data pool a3 are merged, the total number of first type of data finally obtained is also less than or equal to 50. For another example, the sampling number threshold corresponding to the second type of data is set to 100, indicating that the number of the second type of data that needs to be sampled is less than or equal to 100, and the sampling number thresholds corresponding to the data pool b1, data pool b2, and data pool b3 are all 100. Afterwards, after the sampling result B1 corresponding to the data pool b1, the sampling result B2 corresponding to the data pool b2, and the sampling result B3 corresponding to the data pool b3 are merged, the total number of the second type of data that is finally obtained is also less than or equal to 100. For another example, the sampling number threshold corresponding to the third type of data is set to 150, indicating that the number of the third type of data that needs to be sampled is less than or equal to 150, and the sampling number thresholds corresponding to the data pool c1, data pool c2, and data pool c3 are all 150. Afterwards, after the sampling result C1 corresponding to the data pool c1, the sampling result C2 corresponding to the data pool c2, and the sampling result C3 corresponding to the data pool c3 are merged, the total number of the third type of data that is finally obtained is also less than or equal to 150. For details on how the above threads sample each type of data and how different threads merge the sampling results of the same type of data, see the following Figure 2b Description in the corresponding embodiment.

[0118] See also Figure 2b , is a schematic diagram of a thread sampling scenario provided by this application. Figure 2b The described embodiment is a process for how a thread samples a certain type of data read. In other words, Figure 2bThe described embodiment is a specific description of the sampling principle corresponding to a data pool in a thread (corresponding to the sampling of a certain type of data). It can be understood that the sampling principle corresponding to each data pool (a data pool for storing different types in different threads) is the same and independent. Figure 2b As shown, the data pool 100g is assumed to be any data pool in a thread in a sampling device for sampling data (for example, the data pool 100g may be Figure 2a The data pool 100g can store the second type of data read and sampled by thread s3 from the data layer, which can be understood as the data layer storing Figure 2a The data set 100a in the data layer. Thread s3 can read and sample the data in the data layer at the same time. It can be understood that thread s3 continuously reads data from the data layer and stores it in the data pool 100g, and the data pool can filter the stored data (the filtering process is the sampling process). During the filtering process, some data will be retained and some data will be discarded. That is, the method provided by the present application does not need to read all the data and then sample it, but can read and sample at the same time (sampling according to the set sampling data threshold), that is, read and filter at the same time, so that the amount of data stored in the system is limited within the sampling data threshold, thereby saving system capacity and improving sampling efficiency.

[0119] like Figure 2bAs shown, assuming that the sampling number threshold corresponding to the data pool 100g is 5, at the beginning, thread s3 will read data from the data layer with a probability of "1". When the number of read data has not exceeded the sampling data threshold of 5, thread s3 can store all the read data (here, it refers to a certain type of data, because a data pool corresponds to storing a type of data) in the data pool 100g. That is, when thread s3 reads from the first data to the fifth data, it can store the 5 read data in the data pool 100g. At this time, the data stored in the data pool 100g can be 5 data in the data set 101g. When thread s3 reads the sixth data, the data stored in the data pool 100g can be 6 data in the data set 102g. At this time, since the sampling number threshold corresponding to the data pool 100g is 5, and the number of stored data is 6, which exceeds the sampling number threshold 5, it is necessary to sample (that is, filter) the 6 stored data at this time. By sampling the 6 data, part of the 6 data can be discarded. The 6 pieces of data may be sampled at a sampling rate of 1 / 2. After sampling, the data stored in the data pool 100g may be the 3 pieces of data in the data set 103g.

[0120] Next, when reading new data, it is read in at a sampling rate of 1 (that is, the new data is not sampled). Since the newly read data needs to be added to the data pool 100g, that is, the newly read data (that is, assuming that after obtaining the data set 103g, the new data read is data 105g) needs to be merged with the data set 103g that already exists in the data pool 100g, it is necessary to unify the sampling rates between the data to ensure that each data is sampled with equal probability. Specifically, since the data in the data set 103g has been sampled at a sampling rate of 1 / 2, it is also necessary to sample the data 105g at a sampling rate of 1 / 2. Among them, the process of sampling the data in the data set 102g and the data 105g at a sampling rate of 1 / 2 can be seen below. Figure 3 The description of step S102 in the corresponding embodiment. Assume that after sampling the data 105g at a sampling rate of 1 / 2, the data set 104g is obtained, that is, the data stored in the data pool 100g is the data in the data set 104g, that is, on the basis of storing the data in the data set 103g, the data 105g is stored. At this time, the data in the data set 104g are all obtained by sampling at a sampling rate of 1 / 2.

[0121] Next, thread s3 will continue to read data, and thread s3 will also unify the sampling rate of each data read with the data in data set 104g, that is, thread s3 will sample each data read at a sampling rate of 1 / 2. During the process of unifying the sampling rate, some data read and sampled by thread s3 will be added to data pool 100g, and some data read and sampled by thread s3 will be filtered (i.e., discarded) in data pool 100g. When the data in data pool 100g exceeds the sampling quantity threshold 5 again, at this time, the data stored in data pool 100g can be 6 data in data set 106g. The 6 data in data set 106g are all sampled at a sampling rate of 1 / 2, and thread s3 can sample the data in data set 106g at a sampling rate of 1 / 2 again. After sampling the data in the data set 106g at a sampling rate of 1 / 2, the data set 107g can be obtained, that is, at this time, the data stored in the data pool 100g are 3 data in the data set 107g, and the 3 data are sampled from the data set 106g at a sampling rate of 1 / 2. It can be understood that the data in the data set 107g are all sampled at a sampling rate of 1 / 4 (because they have been sampled twice at a sampling rate of 1 / 2).

[0122] Subsequently, thread s3 can still read new data continuously. Similarly, thread s3 will unify the sampling rate of each new data read with the data in data set 107g, that is, thread s3 will sample each new data read with a probability of 1 / 4. In the process of unifying the sampling rate, some new data read and sampled by thread s3 will also be retained in data pool 100g, and some new data read by thread s3 will also be discarded. When the number of data stored in data pool 100g exceeds the sampling number threshold again (that is, the number of data stored in data pool 100g reaches 6 again), at this time, the 6 data stored in data pool 100g are all sampled at a sampling rate of 1 / 4, and thread s3 can sample the data stored in data pool 100g at this time at a sampling rate of 1 / 2 again. After sampling, the number of data stored in data pool 100g can be 3 again, and it can be understood that the 3 data are sampled at a sampling rate of 1 / 8 (because they are sampled 3 times at a sampling rate of 1 / 2). Afterwards, thread s3 may continue to read new data, and cyclically perform the above-described steps on the read new data to obtain a sampling result.

[0123] Finally, when thread s3 no longer reads new data (that is, when sampling is completed), the data pool 100g can store data whose number does not exceed the sampling number threshold of 6, and the final sampling result is obtained. For example, the data stored in the data pool 100g may be 5 pieces of data in the data set 108g, and the sampling rate corresponding to the 5 pieces of data is determined by the number of pieces of data stored in the data pool 100g read by thread s3 and the sampling number threshold corresponding to the data pool 100g. The sampling result corresponding to the data pool 100g is the 5 pieces of data contained in the data set 108g. In the same way as the process described above, it can be obtained. Figure 2a The sampling result A1 corresponds to the data pool a1, the sampling result B1 corresponds to the data pool b1, the sampling result C1 corresponds to the data pool c1, the sampling result A2 corresponds to the data pool a2, the sampling result B2 corresponds to the data pool b2, the sampling result C2 corresponds to the data pool c2, the sampling result A3 corresponds to the data pool a3, the sampling result B3 corresponds to the data pool b3 and the sampling result C3 corresponds to the data pool c3.

[0124] When the sampling results obtained by multiple threads are merged, the principle of merging is the same as the sampling principle corresponding to the above-mentioned thread s3. When there are multiple threads, the sampling results obtained by the multiple threads can be merged in pairs (the merged sampling results are all corresponding to the same type of data). For example, when there are threads 1, 2, 3 and 4, the sampling results of thread 1 and the sampling results of thread 2 can be merged to obtain a merged result 1, and the sampling results of thread 3 and the sampling results of thread 4 can be merged to obtain a merged result 2. Then, the merged result 1 and the merged result 2 can be merged to obtain the total sampling results corresponding to the above-mentioned four threads (including thread 1, thread 2, thread 3 and thread 4). Optionally, the merging method can also be that the sampling results of thread 1 and the sampling results of thread 2 can be merged to obtain a merged result 1. The merged result 1 can be merged with the sampling results of thread 3 to obtain a merged result 3. The merged result 3 can be merged with the sampling results of thread 4 to obtain a merged result 4. The merged result 4 is the total sampling result corresponding to the above-mentioned four threads.

[0125] For example, when the sampling results of thread 1 are merged with the sampling results of thread 2, the sampling rates of the sampling results of thread 1 and thread 2 also need to be unified first. Assuming that the sampling rate corresponding to the multiple data included in the sampling results of thread 1 is 1 / 8, and the sampling rate corresponding to the multiple data included in the sampling results of thread 2 is 1 / 16, then the data included in the sampling results of thread 1 need to be sampled again at a sampling rate of 1 / 2, so that the sampling rates corresponding to the sampling results of thread 1 and the sampling results of thread 2 can be unified to 1 / 16 (because 1 / 8*1 / 2=1 / 16). After the sampling rates corresponding to the multiple data in the sampling results of thread 1 are unified to 1 / 16, some of the multiple data in the sampling results of thread 1 will be discarded. The sampling results of thread 1 with the sampling rate unified to 1 / 16 can be called unified sampling results, and the unified sampling results can be directly merged with the sampling results of thread 2 to obtain a merged sampling result. The merged sampling result includes multiple data (specifically including the data included in the unified sampling result and the data included in the sampling result of thread 2), and the sampling rates of the multiple data in the merged sampling result are all 1 / 16. Then, the number of data in the merged sampling result can be determined. If the number of data in the merged sampling result exceeds the sampling number threshold, the data in the merged sampling result needs to be sampled again at a sampling rate of 1 / 2. After sampling, the final merged result corresponding to thread 1 and thread 2 can be obtained. It can be understood that the sampling rates corresponding to the data in the merged result are all 1 / 32 (i.e., 1 / 16*1 / 2=1 / 32). If the number of data in the above-mentioned merged sampling result does not exceed the sampling number threshold, the merged sampling result can be directly used as the final merged result corresponding to thread 1 and thread 2. When it comes to the merging of the sampling results of multiple threads, the principle of merging is the same as above, that is, the sampling rates are first unified, and after unification, it is determined whether the number of merged data at this time exceeds the sampling data threshold. If it exceeds, sampling is performed again with a probability of 1 / 2 to obtain the final sampling result, ensuring that the sampling rates corresponding to the multiple data in the sampling result obtained by the final merger are the same. If it does not exceed, the merged data can be directly used as the final merge result.

[0126] Through the method provided by the present application, it is supported to sample the read data in parallel during the process of reading a large amount of data, that is, to filter while reading, so that the number of data stored in the data pool can be controlled within the sampling number threshold, saving system capacity and improving sampling efficiency. At the same time, in the process of sampling data, the sampling rate of the read data can be adaptively adjusted according to the sampling number threshold corresponding to the data pool, which is suitable for sampling data in the form of data streams. That is, when the specific number of data to be read is unknown, the sampling rate of the read data can also be adaptively adjusted according to the number of data that need to be sampled in the end, and it is ensured that each data read is sampled at an equal probability sampling rate (here, the same type of data is sampled at the same sampling rate, and different types of data can correspond to different sampling rates), which improves the accuracy of sampling.

[0127] See also Figure 3 , is a flow chart of a data sampling method provided by this application, such as Figure 3 As shown, the method may include:

[0128] Step S101, sampling data of a target type at a first sampling rate in a first time window to obtain target sampled data;

[0129] Specifically, first of all, it is necessary to explain the data sampling process involved in this application. The data that needs to be sampled in this application can be any data, and multiple sampling devices can be used to read and sample the data. Each sampling device can also include multiple threads. Therefore, the final sampling result is that first, each sampling device needs to merge the sampling results corresponding to its multiple threads to obtain the sampling results corresponding to each sampling device, and secondly, it is necessary to merge the sampling results corresponding to each sampling device to obtain the final sampling result. The execution subject in this embodiment can be a sampling system, which includes multiple threads (which can be multiple threads corresponding to multiple sampling devices). The method provided in this application can be applied to the scene of sampling data in a data stream (it can be understood that the data is generated in real time, and the process of sampling the data is also carried out in real time). There can be multiple time windows in the scene, and each time window can correspond to a sampling result. The sampling result corresponding to a certain time window is the result obtained by sampling all the data generated in the time window.

[0130] For example, suppose you need to read and sample the data generated 1 hour ago (or 24 hours ago or other set time length), and sample every 10 minutes to get a sampling result. Then it can be understood that the window time corresponding to each time window is 1 hour (i.e. 60 minutes), the start time points of two adjacent time windows differ by 10 minutes, and the end time points of two adjacent time windows also differ by 10 minutes. See Figure 4 , is a schematic diagram of a time window scenario provided by this application. Figure 4 As shown, the width of the time window can be understood as the length of the time period corresponding to the time window. Since the length of the time period corresponding to the time window (e.g., 1 hour) is usually longer than the time difference between the start times of two adjacent time windows (e.g., 10 minutes), there is usually an intersection time window between two adjacent time windows. The intersection time window refers to the time window corresponding to the same time period in the two time windows. Figure 4 As shown, assuming that the length of each time window is 1 hour, the lengths of time window 100f, time window 101f, time window 102f, time window 103f and time window 104f all represent a time period of 1 hour. Time window 100f and time window 101f are two adjacent time windows, and there is an intersection time window 1 between time window 100f and time window 101f (the intersection time window described in this application refers to the intersection time window of the latter time window compared to the previous time window). Wherein, time window 100f is the first time window, therefore, there is no other time window before time window 100f. The role of time window 100f is to read and sample the data generated within 1 hour before the current moment begins, for example, reading from 13:00, then the data read and sampled in time window 100f is the data generated between 12:00 and 13:00. Since each generated data corresponds to a data generation timestamp, which represents the generation time of each data, when reading data in each time window, the data generated in the corresponding time window can be read according to the data generation timestamp corresponding to each data.

[0131] Among them, the data in each time window can be read and sampled simultaneously and in parallel using multiple threads (which can exist in multiple sampling devices). The above-mentioned time window 101f also includes a time window that does not intersect with the time window 100f, that is, a non-intersecting time window 1. The intersection time window 1 and the non-intersecting time window 1 are added together to obtain the time window 101f. The intersection time window 1 can be called the intersection time window corresponding to the time window 101f, and the non-intersecting time window 1 can be called the non-intersecting time window corresponding to the time window 101f. Similarly, there is also an intersection time window 2 between the time window 101f and the time window 102f. The time window 102f also includes a time window that does not intersect with the time window 101f, that is, a non-intersecting time window 2. The intersection time window 2 and the non-intersecting time window 2 are added together to obtain the time window 102f. The intersection time window 2 can be called the intersection time window corresponding to the time window 102f, and the non-intersecting time window 2 can be called the non-intersecting time window corresponding to the time window 102f. There is also an intersection time window 3 between time window 102f and time window 103f, and time window 103f also includes a time window that does not intersect with time window 102f, that is, non-intersection time window 3. The intersection time window 3 and non-intersection time window 3 are added to obtain time window 103f. Intersection time window 3 can be called the intersection time window corresponding to time window 103f, and non-intersection time window 3 can be called the non-intersection time window corresponding to time window 103f. There is also an intersection time window 4 between time window 103f and time window 104f, and time window 104f also includes a time window that does not intersect with time window 103f, that is, non-intersection time window 4. The intersection time window 4 and non-intersection time window 4 are added to obtain time window 104f. Intersection time window 4 can be called the intersection time window corresponding to time window 104f, and non-intersection time window 4 can be called the non-intersection time window corresponding to time window 104f.

[0132] Since the sampling process corresponding to each time window is serial, the sampling of the data in the next time window will be performed only after the sampling result of the previous time window is obtained. Therefore, the data generated by the timestamp in the intersection time window with the next time window in the sampling result obtained in the previous time window can be directly used as part of the sampling result of the next time window. Therefore, in the next time window, only the new data generated in the time window that does not intersect with the previous time window (i.e., the non-intersection time window) is needed, and the data generated in the intersection time window with the previous time window does not need to be re-read and sampled. This can improve the efficiency of sampling data in a time window. For example, the sampling result corresponding to the above-mentioned time window 100f is the result obtained by reading and sampling all the data from the current moment to the previous 1 hour. After reading and sampling the data generated in the time window 100f to obtain the corresponding sampling result, the sampling process corresponding to the time window 101f can be performed. Assuming that the starting time point of time window 100f differs from that of time window 101f by 10 minutes, when performing the sampling process corresponding to time window 101f, it is only necessary to read and sample the new data generated in non-intersecting time window 1 (10 minutes). After reading and sampling the data generated in non-intersecting time window 1, the data with the data generation timestamp in the intersection time window 1 (50 minutes) in the sampling result corresponding to time window 100f can be directly taken and merged with the sampling result obtained after reading and sampling the data generated in the non-intersecting time window 1, so as to obtain the sampling result corresponding to time window 101f.

[0133] Among them, since the data in each time window is read and sampled in parallel by multiple threads, when executing the sampling process corresponding to a time window, each thread will read and sample the data generated in the time window (the data read by each thread will be different, and the number of data read and sampled by each thread is not certain. From the above, it can be seen that the data read here is the new data generated in the non-intersecting time window corresponding to the time window and the previous time window). Finally, the sampling result corresponding to the time window is obtained by merging the sampling results corresponding to each thread and merging the sampling data obtained in the intersection time window corresponding to the previous time window.

[0134] Among them, for each thread, the data reading process and sampling process are the same, therefore, here, the first thread in the sampling system is taken as an example to explain how to read and sample data, and the first thread can be a thread for reading and sampling data in any sampling device. The data sampled in the first thread can be stored in the sampling database. Moreover, the process of reading and sampling different types of data for each thread is also the same, therefore, here, the first thread is taken as an example to sample the data of the target type (the target type can be any type of the data to be sampled) by the first thread. There can also be multiple sampling databases in the first thread, and one type of sampling data corresponds to one sampling database, that is, one sampling database is used to store the sampled data obtained by sampling a certain type. Therefore, the data of the target type also corresponds to a sampling database, and the sampling database is used to store the sampled data obtained by sampling the data of the target type. The sampling databases in the following process are all described by taking the sampling database corresponding to the data of the target type as an example.

[0135] The data that already exists in the sampling database before the target sampling data is obtained may be called historical sampling data. The historical sampling data may be sampling data obtained by the first thread sampling the target type of data before the target sampling data is obtained. The historical sampling data may be obtained by the first thread itself or sent to it by other threads. In addition, the historical sampling data in the sampling database may also be empty.

[0136] Among them, the process of obtaining the target transaction data can be: each sampled data can include multiple field information. After the first thread reads the original business data (including multiple types of data such as target type data) from the data stream (which can be in the underlying data layer), it will first perform field parsing on the original business data. After parsing, multiple field information of each original business data read can be obtained. The original business data after field parsing can be called initial parsed data (including initial parsed data corresponding to the target type data). The first thread can filter multiple field information in the initial parsed data through a filtering mechanism, that is, a filtering rule can be set in advance. The filtering rule indicates which field information in the initial parsed data needs to be retained and which field information in the initial parsed data needs to be filtered out (i.e. deleted). The filtering operation is an optional option, that is, the multiple field information in the initial parsed data can be filtered, or it can be not filtered, and the multiple field information in the initial parsed data can be retained. The data obtained after filtering the field information of the initial parsed data can be called filtered parsed data (including filtered parsed data corresponding to the target type data). Furthermore, it is also possible to add associated field information to the filter parsing data through the word table association mechanism, that is, the word table that needs to be associated with the word table can be pre-set, and the field information in the word table that needs to be associated with the word table can be added to the filter parsing data. For example, when the field information in the filter parsing data includes personal income information, then through word table association, field information such as personal birthplace can also be added to the filter parsing data. For another example, when the field information in the filter parsing data includes information on whether the individual is married or not, then through word table association, field information such as the name of the individual's parents can also be added to the filter parsing data. The word table association operation is also an option, that is, additional field information can be added to the filter parsing data, or additional field information can not be added to the filter parsing data. The filter parsing data after the word table association operation can be called sampled business data (including sampled business data corresponding to the target type of data). The sampled business data is the data obtained by the first thread from the data layer to complete the processing of the original business data. Subsequently, the sampled business data can be sampled.

[0137] The first thread can identify the data type of the read data through multiple field information in the sampled business data to identify the sampled business data of the target type. The first thread can sample the identified target type data to obtain target sampled data, and the target sampled data can be sampled in real time in the first time window to obtain the sampled data of the target type, and the first time window can be any time window in the sampling process. In the first thread, each type of data (including the target type of data) will correspond to a sampling result, and the sampled data obtained by the sampling result is the data finally stored in the sampling database corresponding to each type of data. The above-mentioned target sampled data can also be the data obtained by the first thread sampling the read data at a first sampling rate (which may be different according to the actual sampling scenario). For example, when new data is read, the target sampled data can also be the new data read by the first thread at a sampling rate of 1 (that is, the first sampling rate is 1).

[0138] In the present application, two sampling scenarios may be included, in which the target sampling data is obtained in different ways. The first sampling scenario of the above two sampling scenarios is: in this sampling scenario, each thread mainly reads the target type data within a time window and performs sampling. Similarly, taking the process in which the first thread samples the target type data (which may be the original data read from the data stream) at the first sampling rate in the first time window to obtain the target sampling data as an example: the first thread will read and sample the data in the data layer. In this stage, a t1 moment can be introduced, and the historical sampling data in the sampling database can be the sampling data of the target type sampled by the first thread before t1 moment. After t1 moment, the first thread can continue to read new data, and the new data read after t1 moment can be called target sampling data (in this special case, the first sampling rate of the target sampling data is 1, that is, the target sampling data is the sampling data of the target type read with a probability of 1). Since in the first sampling scenario, the first thread continuously reads new data, there are historical sampling data and target sampling data for each moment (for example, moment t2) when the first thread reads new data. The historical sampling data is the data of the target type sampled before moment t2, and the target sampling data is the sampling data of the new target type read after moment t2.

[0139] The second sampling scenario in the above two sampling scenarios is: take the first time window and the third time window as an example for explanation, the first time window and the third time window are two adjacent time windows, and the window size of the first time window is equal to that of the second time window. The first time window and the third time window have an intersection time window, and the third time window is the previous time window of the first time window. In other words, the starting time point of the first time window is later than the starting time point of the third time window, and the ending time point of the first time window is also later than the ending time point of the third time window. For example, if a sampling operation is performed every 10 minutes, then the time gap between two adjacent time windows can be 10 minutes, that is, the time difference between the starting time point of the first time window and the starting time point of the second time window can be 10 minutes, and the time difference between the ending time point of the first time window and the ending time point of the second time window can also be 10 minutes.

[0140] In this sampling scenario, the historical sampling data in the sampling database may be sampling data obtained by real-time sampling of data of the target type in the intersection time window between the third time window and the first time window during the third time window, that is, the historical sampling data is sampling data obtained by sampling in the third time window and having a generation time in the intersection time window of the first time window and the third time window. The target sampling data is obtained by sampling the data of the target type at the first sampling rate in the time window of the first time window except the intersection time window with the third time window during the first time window.

[0141] Step S102, merging the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data, wherein a third sampling rate of the merged sampling data is determined according to the first sampling rate and the second sampling rate of the historical sampling data, the historical sampling data is sampling data of the target type obtained by sampling data of the target type before obtaining the target sampling data, the second sampling rate is the sampling rate of the historical sampling data relative to the data of the target type, and the third sampling rate is the sampling rate of the merged sampling data relative to the data of the target type;

[0142] Specifically, the first thread can merge the target sampling data and the historical sampling data in the sampling database to obtain the merged sampling data. Specifically, when the historical sampling data and the target sampling data are the sampling data in the first sampling scenario mentioned above, then the historical sampling data is obtained by sampling the read target type data by the first thread at a certain sampling rate. Assuming that the sampling rate of the historical sampling data is the second sampling rate, the second sampling rate is the sampling rate of the historical sampling data for the target type data. In other words, the second sampling rate is the sampling rate for the original business data (not sampled) corresponding to the historical sampling data. The first sampling rate of the target sampling data is 1. In fact, in this sampling scenario, the target sampling data has not been sampled, but only read. Then, the first thread needs to unify the sampling rates of the target sampling data and the historical sampling data to ensure that the target type data is sampled at the same sampling rate. In this sampling phase, the first sampling rate will definitely be greater than or equal to the second sampling rate (in this application, the sampling rate range is 0-1 as an example for explanation), therefore, the ratio between the second sampling rate and the first sampling rate can be obtained (this ratio can be referred to as the first ratio sampling rate. The first ratio sampling rate range is also between 0 and 1. This ratio is obtained by taking the smaller sampling rate as the dividend and the larger sampling rate as the divisor). The target sampling data can be sampled by this ratio, and the sampling rate of the target sampling data after sampling is the same as the sampling rate of the current historical sampling data. The sampled target sampling data and the current historical sampling data can be directly merged to obtain the merged sampling data. The sampling rate of the merged sampling data can be referred to as the third sampling rate. Therefore, at this time, the third sampling rate of the merged sampling data is equal to the second sampling rate. Among them, the third sampling rate is also the sampling rate relative to the target type of data, that is, the third sampling rate is the sampling rate of the original business data (not sampled) corresponding to the merged sampling data.

[0143] In the present application, the sampling rate can be a power of 1 / 2 (explained in step S103 below), for example, the sampling rate can be 1, ……. For example, if the above-mentioned first sampling rate is 1 / 4, then the above-mentioned first ratio sampling rate is 1 / 4 divided by 1, which is 1 / 4. The target sampling data can be sampled at a sampling rate of 1 / 4. The specific sampling method is: since each data involved in the sampling will have a data identifier (i.e., data ID), the data identifier can be a string of letters and numbers, and the data identifier of each data can be called a data identification string. Therefore, each target sampling data corresponds to a data representation string. Each target transaction data can be mapped to a uniform sampling space through the data identification string of each target transaction data. In this uniform sampling space, each target transaction data can be sampled at the same sampling rate (here 1 / 4). The method of mapping each target transaction data to the uniform sampling space can be to calculate the hash value corresponding to the data identification string of each target transaction data. The target sampling data is sampled at a sampling rate of 1 / 4, that is, the hash value corresponding to each target sampling data is divided by 4. If the division is complete (i.e., the remainder is equal to 0), the corresponding target sampling data can be stored as the sampled data. If the division is not complete (i.e., the remainder is not equal to 0), the corresponding target sampling data can be discarded. After sampling the target sampling data through the above process, the sampled target sampling data can be obtained. In this sampling scenario, the sampled target sampling data and the historical sampling data can be directly merged to obtain the above-mentioned merged sampling data.

[0144] Among them, in the first sampling scenario mentioned above, the second sampling rate of the historical sampling data can also be directly obtained as the first sampling rate (even if the first sampling rate is equal to the second sampling rate), and when reading the new target type of data, the read target type of data is sampled at the first sampling rate to obtain the target sampling data. Therefore, at this time, the sampling rate of the historical sampling data is consistent with the sampling rate of the target sampling data, and the historical sampling data and the target sampling data can be directly merged to obtain the merged sampling data. In this way, the efficiency of merging the target sampling data and the historical sampling data can be improved.

[0145] When the historical sampling data and the target sampling data are data in the second sampling scenario above: that is, the historical sampling data is sampled during the third time window, in the intersection time window of the third time window and the first time window, and the target sampling data is sampled during the first time window, in the time window of the first time window except the intersection time window with the third time window. Similarly, in the second sampling scenario, the target sampling data and the historical sampling data also need to be merged. First, the sampling rates of the target sampling data and the historical sampling data are unified. In this case, the size of the sampling rate between the first sampling rate and the second sampling rate is determined according to the actual sampling scenario. When the first sampling rate corresponding to the target sampling data is less than the second sampling rate corresponding to the historical sampling data, the sampling rate of the historical sampling data needs to be unified from the second sampling rate to the first sampling rate. For example, when the first sampling rate is 1 / 16 and the second sampling rate is 1 / 4, it means that the historical sampling data is obtained by a sampling rate of 1 / 4 and the target sampling data is obtained by a sampling rate of 1 / 16. Theoretically, the historical sampling data needs to be sampled at a sampling rate of 1 / 4 (because the first sampling rate 1 / 16 divided by the second sampling rate 1 / 4 equals 1 / 4, the ratio between the first sampling rate and the second sampling rate can be called the second ratio sampling rate).

[0146] Among them, when the historical sampling data is sampled at the second sampling rate, the hash value corresponding to the data identification string of the historical sampling data is divided by 4. If the remainder is 0, the historical sampling data is stored (the remainder is shown as 0 here because the historical sampling data is stored in the sampling database). If the remainder is not 0, the historical sampling data is discarded. Since the historical sampling data is obtained by sampling at a sampling rate of 1 / 4, the result of dividing the hash value corresponding to the data identification string of the historical sampling data by 4 can be called the first sampling hash value. If the sampling system stores the first sampling hash value when obtaining the historical sampling data, then when the historical sampling data is sampled again at a sampling rate of 1 / 4, the first sampling hash value can be divided by 4. If the remainder is 0, the corresponding historical sampling data is stored. If the remainder is not 0, the corresponding historical sampling data is discarded. If the first sampling hash value is not stored in the sampling system, the steps in step box 100b (i.e., unified sampling rate) can be executed, that is, the historical sampling data is directly sampled using the first sampling rate 1 / 16, that is, the hash value corresponding to the data identification string of the historical sampling data is directly divided by 16, and if the remainder is 0, the corresponding historical sampling data is stored, and if the remainder is not 0, the corresponding historical sampling data is discarded. Through the above process, the sampled historical sampling data can be obtained.

[0147] Since there are usually multiple historical sampling data, some of the historical sampling data can be filtered out by sampling the historical sampling data again. The sampled historical sampling data and the target sampling data can be directly merged to obtain merged sampling data.

[0148] Similarly, in the second sampling scenario, if the first sampling rate is greater than the second sampling rate, the sampling rate of the target sampling data needs to be unified from the first sampling rate to the second sampling rate. For example, when the first sampling rate is 1 / 4 and the second sampling rate is 1 / 16, the target sampling data needs to be sampled again by 1 / 4. As described above, the process of sampling the target sampling data again by 1 / 4 can be to directly divide the hash value corresponding to the data identification string of the target sampling data by 16. If the remainder is 0, the corresponding target sampling data is stored. If the remainder is not 0, the corresponding target sampling data is discarded. If, after sampling the target sampling data using the first sampling rate, the hash value corresponding to the data identification string of the target sampling data divided by 4 is stored, when the target sampling data is sampled again by 1 / 4, the value can be used to divide by 4 again. If the remainder is 0, the corresponding target sampling data is stored. If the remainder is not 0, the corresponding target sampling data is discarded. Since there are usually multiple target sampling data, after sampling the target sampling data again at a sampling rate of 1 / 4, some of the target sampling data can be filtered out. The sampled target sampling data can be obtained through the above process. Similarly, the sampled target sampling data and the historical sampling data can be directly merged to obtain merged sampling data. When the first sampling rate is equal to the second sampling rate, the target sampling data and the merged sampling data can be directly merged to obtain merged sampling data.

[0149] Optionally, in the second sampling scenario, if the number of historical sampling data is less than the merge quantity threshold (which can be set by yourself, for example, set to 10,000 data), and the ratio between the second sampling rate corresponding to the historical sampling data and the first sampling rate corresponding to the target sampling data (the second sampling rate divided by the first sampling rate) is less than the ratio threshold (which can be set by yourself, for example, 1 / 512), then the historical sampling data can be directly discarded and only the target sampling data is retained. In this case, it is necessary to restart accumulating sampling data from the non-intersecting time window corresponding to the second time window, and discard all the data obtained by previous sampling. In this way, it can be avoided that when the data sampled in the current time window (here is the third time window) is too small and the sampling rate is too low, it affects the sampling process in the subsequent time window, for example, causing the number of sampling data obtained by subsequent sampling to be too small and the sampling rate to be too low.

[0150] It can be known from the above process that the third sampling rate corresponding to the obtained combined sampling data is actually the minimum value between the first sampling rate and the second sampling rate.

[0151] See also Figure 5 , is a schematic diagram of a scenario for obtaining combined sampling data provided by this application. Figure 5 As shown, there is an intersection time window between the first time window 101e and the third time window 100e, and the first time window 101e also includes a non-intersecting time window that does not intersect with the third time window 100e. The sampling result 102e is obtained by sampling all the data generated in the third time window 100e. Figure 5 As shown, the sampling result 102e includes sampling data 1, sampling data 2, sampling data 3, sampling data 4, sampling data 5, and sampling data 6. Each sampling data in the sampling result 102e corresponds to a data generation timestamp, and the data generation timestamp represents the generation time of the sampling data. Specifically, the data generation timestamp of sampling data 1 is data generation timestamp 1, the data generation timestamp corresponding to sampling data 2 is data generation timestamp 2, the data generation timestamp corresponding to sampling data 3 is data generation timestamp 3, the data generation timestamp corresponding to sampling data 4 is data generation timestamp 4, the data generation timestamp corresponding to sampling data 5 is data generation timestamp 5, and the data generation timestamp corresponding to sampling data 6 is data generation timestamp 6.

[0152] The process of obtaining historical sampling data from the sampling result 102e may be: the data generation timestamp 1 corresponding to the above sampling data 1 may be recorded as t1, the data generation timestamp 2 corresponding to the above sampling data 2 may be recorded as t2, the data generation timestamp 3 corresponding to the above sampling data 3 may be recorded as t3, the data generation timestamp 4 corresponding to the above sampling data 4 may be recorded as t4, the data generation timestamp 5 corresponding to the above sampling data 5 may be recorded as t5, and the data generation timestamp 6 corresponding to the above sampling data 6 may be recorded as t6. Figure 5As shown in the timestamp schematic box 104e, the data generation timestamp t4 and the data generation timestamp t6 are within the non-intersection time window corresponding to the third time window 100e, and the data generation timestamp t1, the data generation timestamp t2, the data generation timestamp t3 and the data generation timestamp t5 are within the intersection time window corresponding to the third time window 100e. Since the historical sampling data here is the data generation timestamp in the third window sampling data within the intersection time window, the above sampling data 1, sampling data 2, sampling data 3 and sampling data 5 can be used as historical sampling data, that is, the sampling data in the data set 105e is the historical sampling data. In addition, the sampling rate corresponding to each historical sampling data in the data set 105e is the second sampling rate, that is, the sampling rate corresponding to the sampling data 1, sampling data 2, sampling data 3 and sampling data 5 is the second sampling rate.

[0153] Furthermore, in the sampling process corresponding to the first time window 101e, the sampling result obtained by sampling the data generated in the non-intersecting time windows may be the sampling result 103e. At this time, the sampling data in the sampling result 103e may be used as the target sampling data. Figure 5 As shown, it is assumed that the sampling result 103e here includes sampling data 7, sampling data 8 and sampling data 9. And the sampling rate corresponding to sampling data 7, sampling data 8 and sampling data 9 is the first sampling rate. If it is necessary to obtain the final sampling result within the first time window 101e, it is necessary to merge the above-mentioned target sampling data and historical sampling data. When merging, it is necessary to unify the sampling rates of the target sampling data and the historical sampling data. Figure 5 As shown in the data set 106e in FIG. 1 , after unifying the sampling rates between the target sampling data and the historical sampling data, there are sampling data 1, sampling data 7, sampling data 8, and sampling data 9 left, and the sampling data 1, sampling data 7, sampling data 8, and sampling data 9 are the merged sampling data. Moreover, the sampling rates corresponding to the sampling data 1, sampling data 7, sampling data 8, and sampling data 9 are the third sampling rates (the third sampling rates may be equal to the first sampling rates mentioned above), that is, the sampling rates of the merged sampling data are the third sampling rates.

[0154] It can be understood that the sampling results for the target type of data finally obtained in each time window can be obtained after merging the sampling data of the target type sampled by multiple threads in each time window. Among them, the merging of the sampling results of the target type of data of each thread can be performed by any one or more threads. Therefore, the target transaction data obtained by sampling the target type of data at the first sampling rate in the first time window can be sampled by the first thread. The first thread can give the sampled target transaction data to the second thread in the sampling system, and the second thread merges the target sampling data and the historical sampling data to obtain the merged sampling data. It can be seen that in the sampling process, the sampling work is completed by multiple threads in the sampling system in collaboration, which ensures the efficiency of sampling.

[0155] Step S103, when the amount of the combined sampling data is greater than the sampling amount threshold, sampling the combined sampling data based on the adaptive sampling parameter to obtain updated historical sampling data, the sampling amount threshold is related to the data of the target type, and the adaptive sampling parameter is used to control the amount of the combined sampling data within the sampling amount threshold;

[0156] Specifically, when it is detected that the number of the above-mentioned combined sampling data is greater than the sampling number threshold, the combined sampling data can be sampled again by adaptive sampling parameters to obtain updated historical sampling data. Different types of data can correspond to different sampling number thresholds. The sampling number threshold corresponding to a certain type of data indicates that the number of sampled data of this type of data obtained by the final sampling will not exceed (be less than or equal to) its corresponding sampling number threshold. Therefore, the target type of data also corresponds to a sampling number threshold.

[0157] Among them, the above-mentioned adaptive sampling parameter is used to control the number of merged sampled data within the sampling number threshold, and the adaptive sampling parameter can be taken as 1 / 2. The reason is: here, the sampling process corresponding to the target type of data is still used as an example for explanation. Since in the process of sampling the target type of data, the number of sampled data sampled each time will be controlled within the sampling number threshold corresponding to the target type, therefore, the number of the above-mentioned target sampled data and the number of historical sampled data will be less than the sampling number threshold. Then, assuming that the sampling number threshold corresponding to the target type of data is M, it can be known that after merging the target transaction data and the historical sampled data (it can be merged after sampling or directly merged), the number of merged sampled data obtained will be less than or equal to 2 times the sampling number threshold (i.e., less than 2M), because the sum of 2 values ​​less than or equal to M must be less than or equal to 2M. Therefore, taking the adaptive sampling parameter as 1 / 2, and then sampling the merged sampled data by 1 / 2 through the adaptive sampling parameter, the number of merged sampled data after sampling will be less than M. Among them, the merged sampling data is sampled again at a sampling rate of 1 / 2, that is, half of the sampling data in the merged sampling data is filtered out, and half of the sampling data in the merged sampling data is retained to obtain updated historical sampling data (that is, the merged sampling data after sampling). From the above, it can be seen that the number of updated historical sampling data will definitely be less than the sampling number threshold M.

[0158] Among them, the sampling rate of the updated historical sampling data can be called the fourth sampling rate, and the fourth sampling rate is the sampling rate of the updated historical sampling data relative to the data of the target type. In other words, the fourth sampling rate is the sampling rate of the updated historical sampling data for the corresponding original business data (which has not been sampled). The fourth sampling rate is equal to the product of the third sampling rate of the merged sampling data and the adaptive sampling parameter. For example, the third sampling rate is 1 / 8, and the adaptive sampling parameter is 1 / 2, then the fourth sampling rate is 1 / 16. The merged sampling data can be sampled at the fourth sampling rate to obtain the updated historical sampling data. For example, when the fourth sampling rate is 1 / 16, the hash value corresponding to the data identification string of the merged sampling data can be divided by 16. If the remainder is 0, the corresponding merged sampling data is retained. If the remainder is not 0, the corresponding merged sampling data is discarded.

[0159] More, the updated historical sampling data obtained above can be the sampling result corresponding to the first time window. If the second time window is the next time window of the first time window, the sampling data obtained by sampling in the intersection time window between the first time window and the second time window in the updated historical sampling data can be used as part of the sampling data in the sampling result corresponding to the second time window. Therefore, for the sampling process in the second time window, the historical sampling data in the second time window can be the sampling data obtained by sampling in the intersection time window between the first time window and the second time window in the first time window. The target sampling data in the second time window can be the sampling data of the target type sampled in the time window other than the intersection time window between the second time window and the first time window. Therefore, when reading the new target type data in the time window other than the intersection time window between the second time window and the first time window, and sampling the read target type data, the fourth sampling rate (the sampling rate of the updated historical sampling data in the first time window) can be directly used as the first sampling rate in the second time window to sample the read new target type data to obtain the target sampling data. In this way, in the second time window, since the target sampling data obtained by reading the sampling has the same sampling rate as the historical sampling data in the second time window, they can be directly merged to obtain the merged sampling data in the second time window, which improves the efficiency of merging the target sampling data and the historical sampling data in the second time window, and saves system resources at the same time, because there is no need to sample the target sampling data or the historical sampling data and then merge them, and in this way, adaptive adjustment of the sampling rate is achieved. Afterwards, the updated historical sampling data in the second time window can be obtained through the merged sampling data in the second time window, and the updated historical sampling data in the second time window can be the sampling result corresponding to the second time window.

[0160] Step S104, replacing the historical sampling data in the sampling database with updated historical sampling data;

[0161] Specifically, the historical sampling data in the sampling database may be replaced with the updated historical sampling data obtained above. At this time, the updated historical sampling data in the sampling database is the final sampling data obtained by merging the target sampling data and the historical sampling data.

[0162] Among them, if the number of the above-mentioned merged sampling data is less than or equal to the sampling number threshold, there is no need to sample the merged sampling data again, and the historical sampling data in the sampling database can be directly replaced with the merged sampling data. At this time, the merged sampling data in the sampling database is the final sampling data obtained by merging the target sampling data and the historical sampling data. Therefore, through the method provided by the present application, the number of sampling data finally stored in the sampling database can be controlled within the sampling number threshold through the sampling number threshold and the adaptive sampling parameter, and in this process, the sampling rate of the sampling data is also adaptively changed according to the corresponding sampling number threshold and the adaptive sampling parameter.

[0163] See also Figure 6 , is a flow chart of obtaining and updating historical sampling data provided by this application. Figure 6 As shown, when the first sampling rate corresponding to the target sampling data is less than the second sampling rate corresponding to the historical sampling data, step 100b can be executed, that is, the historical sampling data can be sampled using the first sampling rate to obtain the historical sampling data after sampling. Afterwards, step 101b can be executed, that is, the historical sampling data after sampling and the target sampling data are merged to obtain merged sampling data. When the total number of merged sampling data is greater than the sampling number threshold, step 102b can be executed (capacity control, that is, the number of data stored in the sampling database is controlled by the sampling number threshold), that is, the merged sampling data can be sampled by the sampling rate increment coefficient. In this application, the sampling rate increment coefficient is 1 / 2, that is, the merged sampling data can be sampled again by 1 / 2 to obtain updated historical sampling data. Please refer to Figure 7 , is another flowchart of obtaining and updating historical sampling data provided by this application. Figure 7 As shown, when the first sampling rate corresponding to the target sampling data is greater than the second sampling rate corresponding to the historical sampling data, step 100c can be executed, that is, the target sampling data can be sampled using the second sampling rate to obtain the sampled target sampling data. After that, step 101c can be executed, that is, the sampled target sampling data and the historical sampling data are merged to obtain the merged sampling data. When the total number of merged sampling data is less than or equal to the sampling number threshold, step 102b can be executed, that is, the merged sampling data is directly used as the updated historical sampling data.

[0164] Through the above process, it can be known that for each time window, there will be a corresponding sampling result. If the window length of the time window is 60 minutes, the sampling result corresponding to each time window is the result obtained by sampling the data generated in the 60-minute time period corresponding to the time window. The latter time window can directly use the data sampled in the intersection time window obtained in the previous time window. The data plus the data contained in the sampling result obtained by the latter time window in the corresponding non-intersection time window is the sampling result corresponding to the latter time window. The sampling results obtained in each time window can be output and displayed on the user terminal. Each time window corresponds to a sampling state (characterized by the sampling result). In this way, the changes in the sampling results obtained by the previous and next time windows can be intuitively viewed as the time window passes. The sampling results corresponding to each time window output can include sampling results corresponding to multiple different types of data, as well as the number of data obtained by sampling each data and the sampling rate for sampling each data. Different types of data may correspond to different sampling rates and sampling quantity thresholds, but the same type of data must be sampled with equal probability (ie, equal sampling rate, that is, the same sampling rate).

[0165] See also Figure 8 , is another schematic diagram of a data sampling scenario provided by this application. Figure 8As shown, the computing layer includes multiple sampling devices, and each sampling device may include multiple threads. The multiple sampling devices can read the original business data in the data layer, that is, the data layer can provide data for sampling. The data layer can provide several types of data that need to be sampled, specifically including click logs, exposure logs, effect logs, advertising word lists, order word lists, resource word lists and other types of sampling data. The above-mentioned multiple sampling devices can read and sample the data in the data layer in sequence according to the time sequence of each time window (including time window 1, time window 2, time window 3, ..., time window n). After the sampling result corresponding to the previous time window is output, the sampling process corresponding to the next time window can be performed. Time windows may intersect with each other. For example, when the window length of the time window is 60 minutes (i.e., 1 hour), the time window shift time is 10 minutes, that is, global sampling is performed every 10 minutes, then the sampling time period corresponding to time window 1 may be 12:00-13:00 (i.e., the data involved in the sampling is generated between 12:00-13:00), the sampling time period corresponding to time window 2 may be 12:10-13:10, the sampling time period corresponding to time window 3 may be 12:20-13:20, and so on. The intersection time period between the above time window 1 and time window 2 is 12:10-13:00. The sampling result corresponding to each time window can be obtained by sampling the data generated in each time window. Among them, the process of the sampling device reading and sampling the original business data in the data layer can be roughly as follows: First, the sampling device can perform format parsing (i.e., data access format parsing, that is, parsing out the field information in the original business data) on the original business data read from the data layer. Then, the multiple field information in the parsed original business data can be filtered through the filter, that is, the unnecessary field information is filtered out (that is, the data filtering field selection can be filtered through the preset filtering rules). Then, the read data is pre-aggregated through the aggregation function, that is, the data belonging to the same type is classified together. Then, the field information that needs to be added can be added to the multiple field information in the parsed original business data through the vocabulary association operation (it can be added according to the preset addition rules). Next, logical calculation is performed, and logical calculation is the process of sampling the read data after field filtering and vocabulary association. The result output is the sampling result obtained by output sampling, and each time window corresponds to a sampling result. The final sampling result can be stored through the storage layer. Among them, the structure for storing data in the storage layer can include a distributed file system (such as HDFS), a log-type open source database (such as Redis), a relational database (such as MySql) and a structured database (such as Hbase).

[0166] The present application can sample the data of the target type at a first sampling rate in a first time window to obtain the target sampling data; merge the target sampling data and the historical sampling data in the sampling database to obtain the merged sampling data, wherein the third sampling rate of the merged sampling data is determined according to the first sampling rate and the second sampling rate of the historical sampling data, and the historical sampling data is the sampling data of the target type obtained by sampling the data of the target type before obtaining the target sampling data; when the number of the merged sampling data is greater than the sampling number threshold, the merged sampling data is sampled based on the adaptive sampling parameter to obtain the updated historical sampling data, the sampling number threshold is related to the data of the target type, and the adaptive sampling parameter is used to control the number of the merged sampling data within the sampling number threshold; replace the historical sampling data in the sampling database with the updated historical sampling data. It can be seen that the method proposed in the present application can further sample the sampling data when the number of the merged sampling data is greater than the sampling number threshold. In this way, during the sampling period or after the sampling is completed, the number of the sampling data stored in the sampling database can always be controlled within the sampling number threshold range, thereby saving system capacity and realizing adaptive adjustment of the sampling rate.

[0167] See also Fig. 9 , is a schematic diagram of the structure of a data sampling device provided by the present application. The data sampling device may be a computer program (including program code) running in a computer device, for example, the data sampling device is an application software, and the data sampling device may be used to execute the corresponding steps of the method provided in the embodiment of the present application. Fig. 9 As shown, the data sampling device 1 may include: a sampling module 11, a merging module 12, an adaptive sampling module 13 and a replacement module 14;

[0168] A sampling module 11 is used to sample data of a target type at a first sampling rate in a first time window to obtain target sampled data;

[0169] a merging module 12, configured to merge the target sampling data and the historical sampling data in the sampling database to obtain the merged sampling data, wherein the third sampling rate of the merged sampling data is determined according to the first sampling rate and the second sampling rate of the historical sampling data, the historical sampling data is the sampling data of the target type obtained by sampling the data of the target type before obtaining the target sampling data, the second sampling rate is the sampling rate of the historical sampling data relative to the data of the target type, and the third sampling rate is the sampling rate of the merged sampling data relative to the data of the target type;

[0170] An adaptive sampling module 13, configured to sample the merged sampling data based on an adaptive sampling parameter to obtain updated historical sampling data when the amount of merged sampling data is greater than a sampling amount threshold, wherein the sampling amount threshold is related to data of a target type, and the adaptive sampling parameter is configured to control the amount of merged sampling data within the sampling amount threshold;

[0171] The replacement module 14 is used to replace the historical sampling data in the sampling database with updated historical sampling data.

[0172] The specific functional implementation of the sampling module 11, the merging module 12, the adaptive sampling module 13 and the replacement module 14 can be found in Figure 3 Steps S101 to S104 in the corresponding embodiment will not be described in detail here.

[0173] Wherein, the data sampling device 1 is also used for:

[0174] A fourth sampling rate is determined based on the adaptive sampling parameters and the third sampling rate so that the target type of data is sampled using the fourth sampling rate as the first sampling rate in a second time window, wherein the fourth sampling rate is a sampling rate for updating historical sampling data relative to the target type of data, and the second time window is the next time window of the first time window.

[0175] The merging module 12 includes: a first ratio determining unit 121, a first ratio sampling unit 122 and a first merging unit 123;

[0176] A first ratio determination unit 121, configured to determine a ratio between the second sampling rate and the first sampling rate as a first ratio sampling rate when the first sampling rate is greater than the second sampling rate;

[0177] A first ratio sampling unit 122 is used to sample the target sampling data according to a first ratio sampling rate to obtain sampled target sampling data;

[0178] The first merging unit 123 is used to determine the sampled target sampling data and the historical sampling data as merged sampling data.

[0179] The specific functional implementation of the first ratio determination unit 121, the first ratio sampling unit 122 and the first merging unit 123 can be found in Figure 3 Steps S101 to S104 in the corresponding embodiment will not be described in detail here.

[0180] The merging module 12 includes: a second ratio determining unit 124, a second ratio sampling unit 125, a second merging unit 126 and a third merging unit 127;

[0181] A second ratio determination unit 124, configured to determine a ratio between the first sampling rate and the second sampling rate as a second ratio sampling rate when the first sampling rate is less than the second sampling rate;

[0182] A second ratio sampling unit 125 is used to sample the historical sampling data according to a second ratio sampling rate to obtain sampled historical sampling data;

[0183] A second merging unit 126, configured to determine the sampled historical sampling data and the target sampling data as merged sampling data;

[0184] The third merging unit 127 is configured to determine the target sampling data and the historical sampling data as merged sampling data when the first sampling rate is equal to the second sampling rate.

[0185] The specific functional implementation of the second ratio determination unit 124, the second ratio sampling unit 125, the second merging unit 126 and the third merging unit 127 can be found in Figure 3 The corresponding step S102 in the embodiment will not be described in detail here.

[0186] The sampling module 11 includes: a sampling rate determination unit 111 and a window sampling unit 112;

[0187] The sampling rate determining unit 111 is used to determine the second sampling rate of the historical sampling data as the first sampling rate;

[0188] The window sampling unit 112 is used to sample the target type of data using a first sampling rate within a first time window to obtain target sampled data;

[0189] Then, the merge module is also used for:

[0190] The target sampling data and the historical sampling data are determined as merged sampling data.

[0191] The specific functional implementation of the sampling rate determination unit 111 and the window sampling unit 112 can be found in Figure 3 The corresponding step S101 in the embodiment will not be described in detail here.

[0192] The adaptive sampling module 13 includes: a string acquisition unit 131, a mapping unit 132 and a combined sampling unit 133;

[0193] The character string acquisition unit 131 is used to acquire a data identification character string corresponding to the combined sampling data when the amount of combined sampling data is greater than the sampling amount threshold;

[0194] A mapping unit 132, used to map the data identification string into a uniform sampling space to obtain a hash value corresponding to the data identification string;

[0195] The combined sampling unit 133 is used to sample the combined sampling data according to the third sampling rate, the adaptive sampling parameter and the hash value to obtain updated historical sampling data.

[0196] The specific functional implementation of the string acquisition unit 131, the mapping unit 132 and the combined sampling unit 133 can be found in Figure 3 The corresponding step S103 in the embodiment will not be described in detail here.

[0197] The combined sampling unit 133 includes: an adaptive sampling rate subunit 1331 and a combined sampling subunit 1332;

[0198] The adaptive sampling rate subunit 1331 is used to obtain a fourth sampling rate according to the third sampling rate and the adaptive sampling parameter;

[0199] The combined sampling subunit 1332 is used to sample the combined sampling data based on a fourth sampling rate and a hash value to obtain updated historical sampling data; the fourth sampling rate is a sampling rate of the updated historical sampling data relative to the target type of data.

[0200] The specific functional implementation of the adaptive sampling rate subunit 1331 and the combined sampling subunit 1332 can be found in Figure 3 The corresponding step S103 in the embodiment will not be described in detail here.

[0201] The data sampling device 1 is further used for:

[0202] When the amount of the combined sampling data is less than or equal to the sampling amount threshold, the historical sampling data in the sampling database is replaced with the combined sampling data.

[0203] The previous time window of the first time window is the third time window; the first time window and the third time window have an intersection time window; the historical sampling data is the sampling data obtained by sampling the target type of data in the intersection time window of the third time window;

[0204] The sampling module 12 is further specifically used for:

[0205] In a time window other than the intersection time window of the first time window, data of the target type is sampled at a first sampling rate to obtain target sampled data.

[0206] Then, the merging module 12 is further specifically used for:

[0207] When the number of historical sampling data is less than the combined number threshold, and the ratio between the first sampling rate and the second sampling rate is less than the ratio threshold, the historical sampling data is deleted, and the target sampling data is determined as the combined sampling data.

[0208] The sampling module 11 includes: a data reading unit 113, a filtering unit 114, an association unit 115 and a target sampling unit 116;

[0209] The data reading unit 113 is used to read data of a target type from the data stream in a first time window, and perform field parsing on the read data of the target type to obtain initial parsed data;

[0210] A filtering unit 114, configured to filter multiple field information in the initial parsed data based on a filtering mechanism to obtain filtered parsed data;

[0211] An associating unit 115 is used to add associated field information to the filtered and parsed data based on a vocabulary association mechanism to obtain sampled business data;

[0212] The target sampling unit 116 is used to sample the sampled service data at a first sampling rate to obtain target sampled data.

[0213] The specific functional implementation of the data reading unit 113, the filtering unit 114, the association unit 115 and the target sampling unit 116 can be found in Figure 3 The corresponding step S101 in the embodiment will not be described in detail here.

[0214] The sampling module 11 is also specifically used for:

[0215] In a first time window, sampling data of a target type at a first sampling rate through a first thread to obtain target sampled data;

[0216] Then, the merging module 12 is further specifically used for:

[0217] The target sampling data and the historical sampling data in the sampling database are merged through the second thread to obtain merged sampling data.

[0218] The present application can sample the data of the target type at a first sampling rate in a first time window to obtain the target sampling data; merge the target sampling data and the historical sampling data in the sampling database to obtain the merged sampling data, wherein the third sampling rate of the merged sampling data is determined according to the first sampling rate and the second sampling rate of the historical sampling data, and the historical sampling data is the sampling data of the target type obtained by sampling the data of the target type before obtaining the target sampling data; when the number of the merged sampling data is greater than the sampling number threshold, the merged sampling data is sampled based on the adaptive sampling parameter to obtain the updated historical sampling data, the sampling number threshold is related to the data of the target type, and the adaptive sampling parameter is used to control the number of the merged sampling data within the sampling number threshold; replace the historical sampling data in the sampling database with the updated historical sampling data. It can be seen that the method proposed in the present application can further sample the sampling data when the number of the merged sampling data is greater than the sampling number threshold. In this way, during the sampling period or after the sampling is completed, the number of the sampling data stored in the sampling database can always be controlled within the sampling number threshold range, thereby saving system capacity and realizing adaptive adjustment of the sampling rate.

[0219] See also Fig.10 , is a schematic diagram of the structure of a computer device provided by this application. Fig.10 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Fig.10 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0220] exist Fig.10In the computer device 1000 shown in the figure, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to implement the above Figure 3 It should be understood that the computer device 1000 described in this application can also execute the above Fig. 9 The description of the data sampling device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0221] In addition, it should be pointed out here that: the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the data sampling device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3 The description of the data sampling method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0222] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0223] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data sampling method, characterized in that: include: Sampling the data of the target type at a first sampling rate in a first time window to obtain target sampled data; Merging the target sampling data and the historical sampling data in the sampling database to obtain merged sampling data, wherein a third sampling rate of the merged sampling data is determined according to the first sampling rate and a second sampling rate of the historical sampling data, the historical sampling data is sampling data of the target type obtained by sampling data of the target type before obtaining the target sampling data, the second sampling rate is the sampling rate of the historical sampling data relative to the data of the target type, and the third sampling rate is the sampling rate of the merged sampling data relative to the data of the target type; When the amount of the combined sampling data is greater than a sampling amount threshold, sampling the combined sampling data based on an adaptive sampling parameter to obtain updated historical sampling data, wherein the sampling amount threshold is related to the data of the target type, and the adaptive sampling parameter is used to control the amount of the combined sampling data within the sampling amount threshold; The historical sampling data in the sampling database is replaced with the updated historical sampling data.

2. The method according to claim 1, characterized in that The method further comprises: A fourth sampling rate is determined based on the adaptive sampling parameters and the third sampling rate, so that the fourth sampling rate is used as the first sampling rate to sample the data of the target type in a second time window, wherein the fourth sampling rate is the sampling rate of the updated historical sampling data relative to the data of the target type, and the second time window is the next time window of the first time window.

3. The method according to claim 1, characterized in that: The merging of the target sampling data and the historical sampling data in the sampling database to obtain the merged sampling data includes: When the first sampling rate is greater than the second sampling rate, determining a ratio between the second sampling rate and the first sampling rate as a first ratio sampling rate; Sampling the target sampling data according to the first ratio sampling rate to obtain sampled target sampling data; The sampled target sampling data and the historical sampling data are determined as the combined sampling data.

4. The method according to claim 1, characterized in that The merging of the target sampling data and the historical sampling data in the sampling database to obtain the merged sampling data includes: When the first sampling rate is less than the second sampling rate, determining a ratio between the first sampling rate and the second sampling rate as a second ratio sampling rate; Sampling the historical sampling data according to the second ratio sampling rate to obtain sampled historical sampling data; Determine the sampled historical sampling data and the target sampling data as the combined sampling data; When the first sampling rate is equal to the second sampling rate, the target sampling data and the historical sampling data are determined as the combined sampling data.

5. The method according to claim 1, characterized in that The step of sampling the target type of data at a first sampling rate in a first time window to obtain target sampled data includes: Determine the second sampling rate of the historical sampling data as the first sampling rate; In the first time window, sampling the data of the target type using the first sampling rate to obtain the target sampled data; Then, the target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including: The target sampling data and the historical sampling data are determined as the combined sampling data.

6. The method according to claim 1, characterized in that When the amount of the combined sampling data is greater than a sampling amount threshold, sampling the combined sampling data based on an adaptive sampling parameter to obtain updated historical sampling data includes: When the amount of the combined sampling data is greater than the sampling amount threshold, obtaining a data identification character string corresponding to the combined sampling data; Mapping the data identification string to a uniform sampling space to obtain a hash value corresponding to the data identification string; The combined sampling data is sampled according to the third sampling rate, the adaptive sampling parameter and the hash value to obtain the updated historical sampling data.

7. The method according to claim 6, characterized in that The step of sampling the combined sampling data according to the third sampling rate, the adaptive sampling parameter, and the hash value to obtain the updated historical sampling data includes: Obtaining a fourth sampling rate according to the third sampling rate and the adaptive sampling parameter; Based on the fourth sampling rate and the hash value, the merged sampling data is sampled to obtain the update history sampling data; the fourth sampling rate is a sampling rate of the update history sampling data relative to the data of the target type.

8. The method according to claim 1, characterized in that: The method further comprises: When the amount of the combined sampling data is less than or equal to the sampling amount threshold, the historical sampling data in the sampling database is replaced with the combined sampling data.

9. The method according to claim 1, characterized in that: The previous time window of the first time window is a third time window; the first time window and the third time window have an intersection time window; the historical sampling data is the sampling data obtained by sampling the data of the target type within the intersection time window of the third time window; The step of sampling the target type of data at a first sampling rate in a first time window to obtain target sampled data includes: In a time window other than the intersection time window of the first time window, sampling the data of the target type at the first sampling rate to obtain the target sampled data; Then, the target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including: When the number of the historical sampling data is less than the combined number threshold, and the ratio between the first sampling rate and the second sampling rate is less than the ratio threshold, the historical sampling data is deleted, and the target sampling data is determined as the combined sampling data.

10. The method according to claim 1, characterized in that The step of sampling the target type of data at a first sampling rate in a first time window to obtain target sampled data includes: Reading data of the target type from the data stream in the first time window, performing field parsing on the read data of the target type to obtain initial parsed data; Filtering multiple field information in the initial parsed data based on a filtering mechanism to obtain filtered parsed data; Based on the vocabulary association mechanism, the associated field information is added to the filtered and parsed data to obtain the sampled business data; The sampled service data is sampled at the first sampling rate to obtain the target sampled data.

11. The method according to claim 1, characterized in that The step of sampling the target type of data at a first sampling rate in a first time window to obtain target sampled data includes: In the first time window, sampling the data of the target type at the first sampling rate through the first thread to obtain the target sampled data; Then, the target sampling data and the historical sampling data in the sampling database are merged to obtain the merged sampling data, including: The target sampling data and the historical sampling data in the sampling database are merged through a second thread to obtain the merged sampling data.

12. A data sampling device, characterized in that: include: A sampling module, used for sampling data of a target type at a first sampling rate in a first time window to obtain target sampled data; a merging module, configured to merge the target sampling data and the historical sampling data in the sampling database to obtain the merged sampling data, wherein the third sampling rate of the merged sampling data is determined according to the first sampling rate and the second sampling rate of the historical sampling data, the historical sampling data is the sampling data of the target type obtained by sampling the data of the target type before obtaining the target sampling data, the second sampling rate is the sampling rate of the historical sampling data relative to the data of the target type, and the third sampling rate is the sampling rate of the merged sampling data relative to the data of the target type; An adaptive sampling module, used for sampling the merged sampling data based on an adaptive sampling parameter to obtain updated historical sampling data when the amount of merged sampling data is greater than a sampling amount threshold, wherein the sampling amount threshold is related to the data of the target type, and the adaptive sampling parameter is used to control the amount of merged sampling data within the sampling amount threshold; The replacement module is used to replace the historical sampling data in the sampling database with updated historical sampling data.

13. The device according to claim 12, characterized in that The device is also used for: A fourth sampling rate is determined based on the adaptive sampling parameters and the third sampling rate, so that the fourth sampling rate is used as the first sampling rate to sample the target type of data in a second time window, wherein the fourth sampling rate is the sampling rate of the updated historical sampling data relative to the target type of data, and the second time window is the next time window of the first time window.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 11 is executed.

Images