Sampled data real-time updating method and device, equipment and medium

Abstract

The embodiment of the invention discloses a sampling data real-time updating method and device, equipment and a medium. The method comprises the steps of performing data sampling on a target data set based on a pond sampling algorithm to obtain a sampling set; determining target index information of the sampling data according to candidate primary key information and candidate position information of each sampling data in the sampling set; if it is detected that data change occurs in the target data set, determining data change information of the target data set; wherein the data change information comprises target primary key information and corresponding target data; and updating the sampling data in the sampling set according to the data change information and the target index information. According to the technical scheme, when the data set is subjected to data change, the sampling data can be updated in real time based on the constructed index information of the sampling data, and the updating speed and accuracy of the sampling data are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a real-time updating method, device, equipment and medium for sampling data. Background Art

[0002] Using a small amount of sampled data can efficiently generate big data statistics so that the big data computing engine or database can use the optimal execution plan. For example, Spark uses the pond sampling algorithm when partitioning data by range. The pond sampling algorithm ensures that after partitioning by range, the data is evenly distributed in each partition, and there will be no data skew caused by uneven data distribution caused by hash partitioning.

[0003] However, Spark only uses range partitioning when sorting data because it uses the pool sampling algorithm to generate sample data when initiating partitioning, which is very time-consuming. Spark can generate data histograms for each field to understand the data distribution, but data histograms need to be generated in batches by running commands, which is time-consuming, and when the data changes, the data histogram will become inaccurate. If the query conditions are filtered according to other conditions, the data histogram will be invalid. Summary of the invention

[0004] The present invention provides a real-time updating method, device, equipment and medium for sampling data. When data changes occur in a data set, the sampling data can be updated in real time based on the constructed index information of the sampling data, thereby effectively improving the updating speed and accuracy of the sampling data.

[0005] According to one aspect of the present invention, a real-time updating method for sampling data is provided, the method comprising:

[0006] Based on the pond sampling algorithm, data sampling is performed on the target data set to obtain a sampling set;

[0007] Determine target index information of the sampled data according to candidate primary key information and candidate position information of each sampled data in the sampled set;

[0008] If a data change of the target data set is detected, determining data change information of the target data set; wherein the data change information includes target primary key information and corresponding target data;

[0009] The sampled data in the sampled set is updated according to the data change information and the target index information.

[0010] According to another aspect of the present invention, a real-time updating device for sampling data is provided, the device comprising:

[0011] A data sampling module is used to sample the target data set based on a pond sampling algorithm to obtain a sampling set;

[0012] A data index information determination module, used to determine target index information of the sampled data according to candidate primary key information and candidate position information of each sampled data in the sampled set;

[0013] A data change information determination module, configured to determine the data change information of the target data set if a data change is detected in the target data set; wherein the data change information includes target primary key information and corresponding target data;

[0014] The sampling data updating module is used to update the sampling data in the sampling set according to the data change information and the target index information.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and,

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the real-time updating method for sampling data described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the real-time updating method of sampling data described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention is to obtain a sampling set by sampling the target data set based on the pond sampling algorithm; determine the target index information of the sampled data according to the candidate primary key information and candidate position information of each sampled data in the sampling set; if a data change of the target data set is detected, determine the data change information of the target data set; wherein the data change information includes the target primary key information and the corresponding target data; and update the sampled data in the sampling set according to the data change information and the target index information. This technical solution can update the sampled data in real time based on the constructed index information of the sampled data when a data set changes, effectively improving the update speed and accuracy of the sampled data.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a real-time updating method of sampling data provided according to the first embodiment of the present invention;

[0024] Figure 2 is a flow chart of a real-time updating method of sampling data provided according to Embodiment 2 of the present invention;

[0025] Figure 3 is a schematic diagram of the structure of a real-time updating device for sampling data provided according to Embodiment 3 of the present invention;

[0026] Figure 4 The present invention is a schematic diagram of the structure of an electronic device for implementing a real-time updating method of sampling data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 This is a flow chart of a real-time updating method for sampling data provided by the first embodiment of the present invention. This embodiment is applicable to the case of real-time updating of sampling data. The method can be executed by a real-time updating device for sampling data. The real-time updating device for sampling data can be implemented in the form of hardware and / or software. The real-time updating device for sampling data can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0031] S110, sampling the target data set based on a pond sampling algorithm to obtain a sampling set.

[0032] Among them, the pond sampling algorithm is a random sampling algorithm that can be used to extract a representative subset from a large-scale data set for subsequent processing and analysis. It has a wide range of applications in fields such as data analysis and machine learning. The goal of this algorithm is to ensure that each element in the data set has an equal probability of being selected, while only needing to traverse the data set once. The target data set may refer to a data set waiting to be sampled, the target data set includes multiple original data, each original data has unique primary key information (used to uniquely identify the original data, such as ID). The sampling set may refer to a data set obtained by sampling the target data set using the pond sampling algorithm, the sampling set includes multiple sampled data.

[0033] In this embodiment, the target data set is first sampled based on the pond sampling algorithm to obtain a sampling set. The specific process is as follows: 1. Extract the first k original data from the target data set S and put them into the initial set (empty set); 2. Sequentially traverse the target data set S, and for each original data S[i] (i>k) in the target data set S, generate a random number r∈[0,i] according to the subscript i; 3. If r<k, replace the rth sampled data in the initial set with S[i]; 4. After the traversal is completed, the initial set obtained is the sampling set, and all the sampled data in the sampling set are the final sampling results. The above traversal process can be simply understood as, for the i-th (i>k) original data in the target data set S, retain the i-th element according to the probability k / i, and replace it with any one of the k elements of the initial set with the probability of 1 / k. Assuming that the target data set S includes s original data, the sampling probability corresponding to each original data is k / s; and it is necessary to satisfy that the amount of sampled data is less than the amount of original data, that is, k<s.

[0034] It should be noted that the existing data sampling algorithms all sample data sets with unchanged existing data stocks. Therefore, when the amount of data in a data set changes (such as data insertion and data deletion), resampling can only be performed based on the changed data set. Since the amount of data in a data set is usually large, the speed of data sampling is slow. The pond sampling algorithm used in this solution does not pay attention to how much data there is in the data set, and can be applied to situations where the amount of data changes. In addition, the space complexity of the algorithm is only related to the sampling size k, and has nothing to do with the size of the entire data set s. Therefore, the space complexity is low and it can effectively save memory.

[0035] S120, determining target index information of the sampled data according to the candidate primary key information and candidate position information of each sampled data in the sampled set.

[0036] The candidate primary key information may refer to the primary key information corresponding to the sampled data. The candidate position information may refer to the position information of the sampled data in the sampled set. The target index information may be used to characterize the association between the candidate primary key information and the candidate position information.

[0037] In this embodiment, after obtaining the sampling set, the candidate primary key information and candidate position information of each sampled data in the sampling set can be determined respectively, and an index relationship is established between the candidate primary key information and its corresponding candidate position information, thereby obtaining the target index information of each sampled data. It can be understood that the target index information can be used to quickly locate which original data in the target data set the sampled data corresponding to a certain candidate position information comes from.

[0038] S130: If it is detected that a data change occurs in the target data set, data change information of the target data set is determined.

[0039] Among them, data changes may include data insertion, data modification and data deletion. Data change information includes target primary key information and corresponding target data. The target primary key information may refer to the primary key information corresponding to the original data in the target data set where data changes occur. The target data may refer to the data after the original data corresponding to the target primary key information is changed.

[0040] For example, for data insertion changes, suppose that the original data W with primary key information ID1 is added to the target data set, then the target primary key information can be determined to be ID1, and the corresponding target data is W. For data modification changes, suppose that the original data with primary key information ID2 in the target data set is changed from M to N, then the target primary key information can be determined to be ID2, and the corresponding target data is N. For data deletion changes, suppose that the original data V with primary key information ID3 in the target data set is deleted, then the target primary key information can be determined to be ID3, and the corresponding target data is empty.

[0041] S140: Update the sampled data in the sampled set according to the data change information and the target index information.

[0042] In this embodiment, after determining the data change information of the target data set and the target index information of the sampled data, the sampled data in the sampled set can be updated according to the data change information and the target index information. It should be noted that the existing methods for updating sampled data all use a fixed update cycle to update the sampled data, and cannot achieve real-time update of the sampled data, which may result in inaccurate sampled data (i.e., the data in the data set has been updated, but the sampled data has not been updated). In order to ensure the update speed and accuracy of the sampled data at the same time, the target index information of the sampled data is established in this solution, and the real-time update of the sampled data is achieved based on the target index information, that is, when the data of the target data set changes, the sampled data can be updated synchronously, thereby effectively improving the update speed and accuracy of the sampled data.

[0043] With this kind of sampled data that is updated synchronously with the data set, various statistical information can be quickly generated, and the data can be quickly partitioned by range, so that the big data computing engine or database can use the most optimized execution strategy for efficient query; it can be used to quickly analyze the distribution, characteristics and trends of data, and provide approximate results for decision-making and prediction. In some scenarios, the analysis results of sampled data can represent the characteristics of the overall data.

[0044] In this embodiment, optionally, the data change of the target data set includes data insertion; accordingly, the sampled data in the sampling set is updated according to the data change information and the target index information, including: using a pond sampling algorithm to update the sampled data in the sampling set according to the target data in the data change information; determining whether there is sampled data in the sampling set to update it as the target data based on the update result; if so, determining the candidate position information where the target data is located as the target position information; and updating the candidate primary key information corresponding to the target position information to the target primary key information corresponding to the target data.

[0045] Specifically, when a data insertion change occurs in the target data set, the pond sampling algorithm is first used to update the sampled data in the sampling set according to the target data in the data change information to obtain the update result. Among them, if the target data is retained in the sampling set, it is necessary to replace a certain sampled data with the target data to determine the update result to achieve the update of the sampled data; if the target data is not retained in the sampling set, there is no need to replace the sampled data, and the update result is the previous sampling result (that is, no update). Then, based on the update result, it is determined whether there is sampled data in the sampling set that is updated to the target data. If it exists, it indicates that the sampled data has been updated. At this time, the candidate location information where the target data is located can be determined as the target location information, and the candidate primary key information corresponding to the target location information can be updated to the target primary key information corresponding to the target data; if it does not exist, it indicates that the sampled data has not been updated, and no processing is performed at this time.

[0046] Through such a setting, this solution can quickly, accurately and synchronously update the sampled data according to the data change information and the target index information when data insertion changes occur in the target data set.

[0047] The technical solution of the embodiment of the present invention is to obtain a sampling set by sampling the target data set based on the pond sampling algorithm; determine the target index information of the sampled data according to the candidate primary key information and candidate position information of each sampled data in the sampling set; if a data change of the target data set is detected, determine the data change information of the target data set; wherein the data change information includes the target primary key information and the corresponding target data; and update the sampled data in the sampling set according to the data change information and the target index information. This technical solution can update the sampled data in real time based on the constructed index information of the sampled data when a data set changes, effectively improving the update speed and accuracy of the sampled data.

[0048] In this embodiment, optionally, the data change of the target data set includes data modification; accordingly, the sampled data in the sampling set is updated according to the data change information and the target index information, including: matching the target primary key information in the data change information with the candidate primary key information of each sampled data in the sampling set; if the match is successful, determining the target position information corresponding to the target primary key information according to the target index information; and updating the sampled data corresponding to the target position information to the target data in the data change information.

[0049] Specifically, when the target data set undergoes data modification, the target primary key information in the data modification information is first matched with the candidate primary key information of each sampled data in the sampling set to obtain a matching result. If the candidate primary key information contains the same primary key information as the target primary key information, it indicates that the modified original data appears in the sampled data, and at this time, it can be determined that the match is successful; if the candidate primary key information does not contain the same primary key information as the target primary key information, it indicates that the modified original data does not appear in the sampled data, and at this time, it can be determined that the match fails.

[0050] If the match is successful, the target index information corresponding to the target primary key information is determined (that is, the candidate primary key information in the target index information is consistent with the target primary key information), and the candidate position information in the target index information is determined as the target position information, and then the sampling data corresponding to the target position information is updated to the target data in the data change information; if the match fails, no processing is performed.

[0051] Through such a setting, this solution can quickly, accurately and synchronously update the sampled data according to the data change information and the target index information when the target data set is modified.

[0052] Embodiment 2

[0053] Figure 2 A flowchart of a real-time update method for sampling data provided in the second embodiment of the present invention, this embodiment is optimized based on the above embodiment. The specific optimization is as follows: the data change of the target data set includes data deletion; accordingly, the sampling data in the sampling set is updated according to the data change information and the target index information, including: matching the target primary key information in the data change information with the candidate primary key information of each sampling data in the sampling set; if the match is successful, determining the target position information corresponding to the target primary key information according to the target index information; clearing the sampling data corresponding to the target position information, and adding a deletion mark to the target position information, while retaining the candidate primary key information corresponding to the target position information.

[0054] like Figure 2 As shown, the method of this embodiment specifically includes the following steps:

[0055] S210, sampling the target data set based on a pond sampling algorithm to obtain a sampling set.

[0056] S220, determining target index information of the sampled data according to the candidate primary key information and candidate position information of each sampled data in the sampled set.

[0057] S230: If data deletion of the target data set is detected, data change information of the target data set is determined.

[0058] The data change information includes target primary key information and corresponding target data.

[0059] S240: Match the target primary key information in the data change information with the candidate primary key information of each sampled data in the sampling set.

[0060] S250: If the match is successful, determine the target location information corresponding to the target primary key information according to the target index information.

[0061] Among them, the specific implementation method of S210-S250 can refer to the relevant description in the above embodiment, and will not be repeated here.

[0062] S260, clearing the sampled data corresponding to the target location information, adding a deletion mark to the target location information, and retaining the candidate primary key information corresponding to the target location information.

[0063] In this embodiment, when a data deletion change occurs in the target data set, and the deleted original data appears in the sampled data (i.e., the match is successful), it is necessary to determine the target location information corresponding to the target primary key information based on the target index information, and clear the sampled data corresponding to the target location information, but it is necessary to retain the target location information, add a deletion mark to the target location information, and retain the candidate primary key information corresponding to the target location information. Furthermore, if the deleted original data does not appear in the sampled data (i.e., the match fails), no processing is performed. Therefore, the data deletion change can be treated as a special case of the data modification change. The difference between the two is that the data modification change is to modify the sampled data corresponding to the target location information to the target data, and the data deletion change is to modify the sampled data corresponding to the target location information to empty (i.e., clear).

[0064] The purpose of doing this is to ensure that each data in the sampling set is sampled with the same probability, and the sampling probability of each data is k / (n+d), where n is the original data size of the current target data set and d is the deleted data size of the original target data set.

[0065] The technical solution of the embodiment of the present invention is that when the data of the target data set is changed to data deletion, the target primary key information in the data change information is matched with the candidate primary key information of each sampled data in the sampling set; if the match is successful, the target position information corresponding to the target primary key information is determined according to the target index information; the sampled data corresponding to the target position information is cleared, and a deletion mark is added to the target position information, while retaining the candidate primary key information corresponding to the target position information. This technical solution ensures that the sampling probability of each data remains the same by deleting the sampled data but retaining the corresponding position when the target data set undergoes a data deletion change, and can quickly, accurately and synchronously update the sampled data.

[0066] In this embodiment, optionally, after the sampled data corresponding to the target location information is cleared and a deletion mark is added to the target location information, while retaining the candidate primary key information corresponding to the target location information, it also includes: if it is detected that the target data corresponding to the target primary key information is inserted into the target data set again, the sampled data in the sampling set is updated according to the target data using a pond sampling algorithm; based on the update result, it is determined whether the target location information in the sampling set corresponds to the target data; if so, the deletion mark of the target location information is cleared.

[0067] Specifically, after the sampling data is synchronously updated for data deletion changes, if it is detected that the target data corresponding to the target primary key information is inserted into the target data set again, the pond sampling algorithm is first used to update the sampled data in the sampling set according to the target data, and then based on the update result, it is determined whether the target position information in the sampling set corresponds to the target data, that is, whether the target data exists at the target position. If so, it indicates that the target data is reinserted into the target position, and the deletion mark of the target position information needs to be cleared; otherwise, it indicates that the target data is not reinserted into the target position, and the deletion mark of the target position information does not need to be cleared.

[0068] Through such a setting, when the deleted sample data is inserted into the same position again, there is no need to change the target index information of the position, and only the deletion mark needs to be cleared.

[0069] In this embodiment, optionally, after determining whether the target position information in the sampling set corresponds to the target data based on the update result, it also includes: if not, determining whether there is sampling data in the sampling set to update it as the target data based on the update result; if so, determining the candidate position information where the target data is located as the reference position information; and updating the candidate primary key information corresponding to the reference position information to the target primary key information corresponding to the target data.

[0070] In this embodiment, if it is determined based on the update result that the target position information in the sampling set does not correspond to the target data, that is, the target data does not exist at the target position, it is necessary to further determine whether there is sampled data in the sampling set to be updated as the target data based on the update result. If so, it indicates that the sampled data update occurs at other candidate positions other than the target position (this situation is equivalent to data modification). At this time, the candidate position information where the target data is located can be determined as the reference position information, and the candidate primary key information corresponding to the reference position information can be updated to the target primary key information corresponding to the target data; if not, no processing is performed.

[0071] Through such a setting, when the deleted sample data is inserted into another position, the target index information of the position needs to be changed accordingly.

[0072] Embodiment 3

[0073] Figure 3 This is a schematic diagram of the structure of a real-time updating device for sampling data provided by the third embodiment of the present invention. The device can execute the real-time updating method for sampling data provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 3 As shown, the device comprises:

[0074] A data sampling module 310 is used to perform data sampling on a target data set based on a pond sampling algorithm to obtain a sampling set;

[0075] A data index information determination module 320, configured to determine target index information of the sampled data according to candidate primary key information and candidate position information of each sampled data in the sampled set;

[0076] The data change information determination module 330 is used to determine the data change information of the target data set if a data change is detected in the target data set; wherein the data change information includes target primary key information and corresponding target data;

[0077] The sampling data updating module 340 is used to update the sampling data in the sampling set according to the data change information and the target index information.

[0078] Optionally, the data change of the target data set includes data insertion;

[0079] Accordingly, the sampling data updating module 340 is used to:

[0080] Using a pond sampling algorithm to update the sampled data in the sampling set according to the target data in the data change information;

[0081] Determine whether the sampled data exists in the sampled set based on the update result and update it to the target data;

[0082] If so, determining the candidate location information where the target data is located as the target location information;

[0083] The candidate primary key information corresponding to the target location information is updated to the target primary key information corresponding to the target data.

[0084] Optionally, the data change of the target data set includes data modification;

[0085] Accordingly, the sampling data updating module 340 is further used for:

[0086] Matching the target primary key information in the data change information with the candidate primary key information of each sampled data in the sampling set;

[0087] If the match is successful, determining the target location information corresponding to the target primary key information according to the target index information;

[0088] The sampling data corresponding to the target position information is updated to the target data in the data change information.

[0089] Optionally, the data change of the target data set includes data deletion;

[0090] Accordingly, the sampling data updating module 340 is further used for:

[0091] Matching the target primary key information in the data change information with the candidate primary key information of each sampled data in the sampling set;

[0092] If the match is successful, determining the target location information corresponding to the target primary key information according to the target index information;

[0093] The sampled data corresponding to the target location information is cleared, and a deletion mark is added to the target location information, while the candidate primary key information corresponding to the target location information is retained.

[0094] Optionally, the sampling data updating module 340 is further used to:

[0095] After the sampled data corresponding to the target location information is cleared, a deletion mark is added to the target location information, and the candidate primary key information corresponding to the target location information is retained, if it is detected that the target data corresponding to the target primary key information is inserted into the target data set again, the sampled data in the sampling set is updated according to the target data using a pond sampling algorithm;

[0096] Determine whether the target location information in the sampling set corresponds to the target data based on the update result;

[0097] If so, the deletion mark of the target location information is cleared.

[0098] Optionally, the sampling data updating module 340 is further used to:

[0099] After determining whether the target location information in the sampling set corresponds to the target data based on the update result, if not, determining whether the sampling data in the sampling set exists based on the update result and updating it to the target data;

[0100] If so, determining the candidate location information where the target data is located as the reference location information;

[0101] The candidate primary key information corresponding to the reference position information is updated to the target primary key information corresponding to the target data.

[0102] A real-time updating device for sampling data provided by an embodiment of the present invention can execute a real-time updating method for sampling data provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0103] Embodiment 4

[0104] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0105] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0107] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a real-time update method for sampling data.

[0108] In some embodiments, the real-time update method of the sampling data may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the real-time update method of the sampling data described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the real-time update method of the sampling data by any other appropriate means (e.g., by means of firmware).

[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0111] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0114] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0116] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time updating method for sampling data, characterized in that: The method comprises: Based on the pond sampling algorithm, data sampling is performed on the target data set to obtain a sampling set; Determine target index information of the sampled data according to candidate primary key information and candidate position information of each sampled data in the sampled set; If a data change of the target data set is detected, determining data change information of the target data set; wherein the data change information includes target primary key information and corresponding target data; The sampled data in the sampled set is updated according to the data change information and the target index information.

2. The method according to claim 1, characterized in that The data change of the target data set includes data insertion; Correspondingly, updating the sampled data in the sampled set according to the data change information and the target index information includes: Using a pond sampling algorithm to update the sampled data in the sampling set according to the target data in the data change information; Determine whether the sampled data exists in the sampled set based on the update result and update it to the target data; If so, determining the candidate location information where the target data is located as the target location information; The candidate primary key information corresponding to the target location information is updated to the target primary key information corresponding to the target data.

3. The method according to claim 1, characterized in that The data change of the target data set includes data modification; Correspondingly, updating the sampled data in the sampled set according to the data change information and the target index information includes: Matching the target primary key information in the data change information with the candidate primary key information of each sampled data in the sampling set; If the match is successful, determining the target location information corresponding to the target primary key information according to the target index information; The sampling data corresponding to the target position information is updated to the target data in the data change information.

4. The method according to claim 1, characterized in that: The data change of the target data set includes data deletion; Correspondingly, updating the sampled data in the sampled set according to the data change information and the target index information includes: Matching the target primary key information in the data change information with the candidate primary key information of each sampled data in the sampling set; If the match is successful, determining the target location information corresponding to the target primary key information according to the target index information; The sampled data corresponding to the target location information is cleared, and a deletion mark is added to the target location information, while the candidate primary key information corresponding to the target location information is retained.

5. The method according to claim 4, characterized in that After clearing the sample data corresponding to the target location information, adding a deletion mark to the target location information, and retaining the candidate primary key information corresponding to the target location information, the method further includes: If it is detected that the target data corresponding to the target primary key information is inserted into the target data set again, the sampled data in the sampling set is updated according to the target data using a pond sampling algorithm; Determine whether the target location information in the sampling set corresponds to the target data based on the update result; If so, the deletion mark of the target location information is cleared.

6. The method according to claim 5, characterized in that After determining whether the target location information in the sampling set corresponds to the target data based on the update result, the method further includes: If not, determining whether the sampled data exists in the sampled set based on the update result and updating it as the target data; If so, determining the candidate location information where the target data is located as the reference location information; The candidate primary key information corresponding to the reference position information is updated to the target primary key information corresponding to the target data.

7. A real-time updating device for sampling data, characterized in that: The device comprises: A data sampling module is used to sample the target data set based on a pond sampling algorithm to obtain a sampling set; A data index information determination module, used to determine target index information of the sampled data according to candidate primary key information and candidate position information of each sampled data in the sampled set; A data change information determination module, configured to determine the data change information of the target data set if a data change is detected in the target data set; wherein the data change information includes target primary key information and corresponding target data; The sampling data updating module is used to update the sampling data in the sampling set according to the data change information and the target index information.

8. The device according to claim 7, characterized in that The data change of the target data set includes data insertion; Correspondingly, the sampling data updating module is used to: Using a pond sampling algorithm to update the sampled data in the sampling set according to the target data in the data change information; Determine whether the sampled data exists in the sampled set based on the update result and update it to the target data; If so, determining the candidate location information where the target data is located as the target location information; The candidate primary key information corresponding to the target location information is updated to the target primary key information corresponding to the target data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the real-time updating method for sampling data according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the real-time updating method for sampling data according to any one of claims 1 to 6 when executed.

Images