A data sampling method and device based on a distributed memory database

Abstract

The present application relates to a data sampling method based on a distributed memory database, which uses the distributed memory database as a filtering container, and the data filtering rule is a filtering condition. The filtering container attributes include distributed cluster servers, data cache size, and data cache policy. The filtering conditions include calculating a 128-bit HASH value based on a rule according to the MD5 algorithm and constructing a data storage memory database organization rule based on the HASH value, so as to quickly and accurately extract data according to custom rules in the face of large amounts of big data. It can meet the data sampling effect at various PB levels and can meet the effect of obtaining the required sampled result data in a short time without affecting the business efficiency during business use. Through sampling analysis of large amounts of data in various industries, the situation can be quickly grasped, various early warnings and judgments can be made in advance, and it has great significance in actual combat in various fields such as life production, event development, and disaster prediction.

Description

Technical Field

[0001] This application relates to the technical field of data sampling, and particularly to a data sampling method and device based on a distributed memory database. Background Art

[0002] Data sampling has a profound impact on people's lives, reflecting the development trends of things from various dimensions such as time, space, and quantity. Therefore, data sampling practices widely exist in all walks of life to observe the development of things and predict future trends. Currently, there are various sampling methods in the market, and each industry forms a specific sampling method according to its own characteristics, such as manual collection, collection based on questionnaires, collection based on log data generated by business systems, etc. In the current big data era, data sampling from massive data reflects the difficulty, accuracy, and timeliness of collection, and data sampling needs to be carried out in a more optimized manner.

[0003] In the current big data information era, there is a vast amount of data, and it is no longer possible to simply collect and analyze data manually or simply extract data. It is necessary to regularly and accurately extract valuable data samples to reflect the accuracy of the sampled data. Currently, it is difficult to quickly and accurately extract data according to custom rules in the face of massive big data. Summary of the Invention

[0004] In view of the above technical problems, this application proposes a data sampling method and device based on a distributed memory database.

[0005] In a first aspect, this application proposes a data sampling method based on a distributed memory database, including the following steps:

[0006] S1: When the program based on stream processing waits for data to arrive, calculate the HASH value of the current data based on a preset rule;

[0007] S2: Build a storage filtering container: Deploy a database cluster with 80% of the system's available resource pool in the distributed memory database and divide it into multiple sub-nodes;

[0008] S3: Write the HASH value of the current data into the distributed memory database;

[0009] S4: When the next data arrives, calculate the HASH value of this data based on a preset rule, and match it in the distributed memory database according to the HASH value of this data. If the same HASH value exists in the distributed memory database, filter this data; if the same HASH value does not exist in the distributed memory database, store this data in the distributed memory database.

[0010] By adopting the above technical solution, the present invention uses a distributed memory database as a filtering container, and the data filtering rule is a filtering condition. The filtering container attributes include a distributed cluster server, data cache size, and data caching strategy. The filtering condition includes calculating a 128-bit HASH value based on a rule according to the MD5 algorithm, and constructing a data storage memory database organization rule based on the HASH value, so as to quickly and accurately extract data according to a custom rule in the face of large amounts of big data.

[0011] Preferably, the S1 specifically includes:

[0012] S11: When the program based on stream processing waits for data to arrive, calculate the file size of the current record, and distinguish data greater than 10240KB and less than or equal to 10240KB;

[0013] S12: For data greater than 10240KB, intercept the first 1024KB and the last 1024KB of the file for MD5 calculation. For data less than or equal to 10240KB, convert it to byte type and perform full data reverse order calculation, and then perform MD5 full data calculation, so as to calculate the HASH value of the current data.

[0014] Preferably, in the S3, after writing the HASH value of the current data into the distributed memory database, assign a cache expiration time.

[0015] Preferably, in the S4, if there is no same HASH value in the distributed memory database, store this data into the distributed memory database, and load this data into the memory and assign a cache expiration time.

[0016] In a second aspect, the present application also proposes a data sampling device based on a distributed memory database. The device includes:

[0017] HASH value calculation module, configured to calculate the HASH value of the current data based on a preset rule when the program based on stream processing waits for data to arrive;

[0018] Storage filtering container construction module, configured to deploy a database cluster according to 80% of the system available resource pool for the distributed memory database, and divide it into multiple sub-nodes;

[0019] HASH value storage module, configured to write the HASH value of the current data into the distributed memory database;

[0020] A data filtering module, when the next piece of data arrives, calculates the HASH value of this piece of data based on a preset rule, and matches it in the distributed memory database according to the HASH value of this piece of data. If there is the same HASH value in the distributed memory database, this piece of data is filtered. If there is no same HASH value in the distributed memory database, this piece of data is stored in the distributed memory database.

[0021] Preferably, when the program based on stream processing waits for data to arrive and calculates the HASH value of the current data based on a preset rule, it specifically includes:

[0022] When the program based on stream processing waits for data to arrive, it calculates the file size of the current record, and differentiates the data greater than 10240KB and less than or equal to 10240KB;

[0023] For the data greater than 10240KB, the first 1024KB and the last 1024KB of the file are intercepted for MD5 calculation. For the data less than or equal to 10240KB, it is converted into byte type and then the full data is reversed and calculated, and then the MD5 full data calculation is performed, so as to calculate the HASH value of the current data.

[0024] Preferably, after the HASH value storage module writes the HASH value of the current data into the distributed memory database, it assigns a cache expiration time.

[0025] Preferably, in the data filtering module, if there is no same HASH value in the distributed memory database, this piece of data is stored in the distributed memory database, and this piece of data is loaded into the memory and assigned a cache expiration time.

[0026] In a third aspect, the present application also proposes an electronic device, including:

[0027] One or more processors;

[0028] A storage device for storing one or more programs;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.

[0030] In a fourth aspect, the present application also proposes a computer-readable storage medium, on which a computer program is stored, and characterized in that when the program is executed by a processor, it implements the method described in the first aspect.

[0031] The present application at least includes the following beneficial technical effects:

[0032] 1. The present invention uses a distributed memory database as a filtering container, and the data filtering rule is a filtering condition. The filtering container attributes include a distributed cluster server, data cache size, and data cache policy. The filtering condition includes calculating a 128-bit HASH value based on a rule using the MD5 algorithm and constructing a data storage memory database organization rule based on the HASH value, so as to quickly and accurately extract data according to a custom rule in the face of big data and massive data;

[0033] 2. The present invention mainly uses a distributed memory database as a container to quickly filter and extract massive data result information, which can meet the data sampling effects of various PB levels, and can meet the requirement of obtaining the required sampled result data in a short time without affecting the business efficiency during business use. Through sampling analysis of massive data in various industries, the situation can be quickly grasped, and various early warnings and judgments can be made in advance, which has great significance in actual combat in various fields such as life production, the development of things, and disaster prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The drawings illustrate the embodiments and, together with the description, are used to explain the principles of the present application. Other embodiments and many of the expected advantages of the embodiments will be readily recognized as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. Like reference numerals refer to corresponding like parts.

[0035] Figure 1 is a flowchart of a data sampling method based on a distributed memory database according to an embodiment of the present application.

[0036] Figure 2 is a schematic diagram of a specific embodiment that can be applied to the data sampling method based on a distributed memory database according to the present application.

[0037] Figure 3 is a specific flowchart diagram of step S1 of the data sampling method based on a distributed memory database in an embodiment of the present application.

[0038] Figure 4 is a schematic diagram of the MD5 algorithm in an embodiment of the present application.

[0039] Figure 5 is a schematic diagram of the module structure of a data sampling device based on a distributed memory database in an embodiment of the present application.

[0040] Figure 6 is a schematic diagram of the structure of a computer system of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than limiting the invention. In addition, it should be noted that for the sake of convenience of description, only the parts related to the relevant invention are shown in the drawings.

[0042] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0043] Figure 1 The flowchart of a data sampling method based on a distributed memory database according to the present application is shown. Figure 2 The schematic diagram of a specific embodiment of the data sampling method based on a distributed memory database that can be applied to the present application is shown. With reference to Figure 1 and Figure 2 , the method specifically includes the following steps:

[0044] S1: When the program based on stream processing waits for data to arrive, calculate the HASH value of the current data based on a preset rule;

[0045] In an alternative embodiment, the execution of S1 can be completed according to the following steps:

[0046] S11: When the program based on stream processing waits for data to arrive, calculate the file size of the current record, and distinguish the data greater than 10240KB and less than or equal to 10240KB;

[0047] S12: For the data greater than 10240KB, calculate the MD5 of the first 1024KB and the last 1024KB of the intercepted file. For the data less than or equal to 10240KB, convert it to byte type and then perform full-data reverse calculation, and then perform MD5 full-data calculation, so as to calculate the HASH value of the current data.

[0048] In a specific embodiment, through the reduction and analysis of the PB-level traffic of network data, it is obtained that 99.6% of the data greater than 10240KB contains various attachments, such as images, audio, and video, and most of the data less than or equal to 10240KB is text data. Therefore, setting the boundary at 10240KB can meet the actual usage scenarios.

[0049] Referring to Figure 3 , for the data greater than 10240KB, calculate the MD5 of the first 1024KB and the last 1024KB of the intercepted file. Among them, the MD5 calculation logic is as Figure 4As shown, the MD5 encryption process first defines four values, calculates the original text using these four values, and then obtains four new values. This process is repeated a certain number of times, and finally, the four values are concatenated into a string to obtain the final ciphertext.

[0050] Convert data less than or equal to 10240KB to byte type and then perform full-data reverse calculation. The calculation example process of full-data reverse calculation is as follows:

[0051] First, group the data two by two, swap the first half and the second half, then group the data four by four, swap the first half and the second half, and then group the data eight by eight, swap the first half and the second half.

[0052] For example, reverse 12345678 to 87654321:

[0053] First, group the data two by two and swap the first half and the second half to get 21436587. Then, group the data four by four and swap the first half and the second half to get 43218765. Then, group the data eight by eight and swap the first half and the second half to get 87654321.

[0054] The code is as follows:

[0055] static byte ReverseBits(byte c)

[0056] {

[0057] c = (byte)(((byte)(c & 0x55) << 1)) | ((byte)((c & 0xAA) >> 1)));

[0058] c = (byte)(((byte)(c & 0x33) << 2)) | ((byte)((c & 0xCC) >> 2)));

[0059] c = (byte)(((byte)(c & 0x0F) << 4)) | ((byte)((c & 0xF0) >> 4)));

[0060] return c

[0061] }

[0062] After converting data less than or equal to 10240KB to byte type and performing full-data reverse calculation, perform MD5 full-data calculation on the result. Reverse calculation makes the data distribution cluster more balanced, facilitating faster and more efficient calculation of large-capacity stored data.

[0063] S2: Construct a storage filtering container: Deploy a database cluster of the distributed in-memory database according to 80% of the system's available resource pool, and divide it into multiple sub-nodes;

[0064] In a specific embodiment, deploy the storage filtering container, i.e., the distributed in-memory database, according to 80% of the system's available resource pool to form a database cluster, and divide it into multiple sub-nodes to establish a cluster. The advantages of the multiple sub-node clusters are that the data is stored and distributed among multiple nodes according to slots, data is shared among nodes, data distribution can be dynamically adjusted, scalability is high, there is no central architecture, high availability is achieved, operation and maintenance costs can be reduced, and the availability and scalability of the system can be effectively improved.

[0065] S3: Write the HASH value of the current data into the distributed in-memory database;

[0066] In a specific embodiment, in S3, after writing the HASH value of the current data into the distributed in-memory database, assign a cache expiration time. Assigning the cache expiration time is also assigning the sampling period time. Its function is to be valid within a sampling time period. When the data exceeds a sampling time period, the data will automatically expire and be deleted. After expiration, the data will no longer be matched and hit, thus ensuring the accuracy of the sampled data.

[0067] S4: When the next piece of data arrives, calculate the HASH value of this piece of data based on a preset rule, and match it in the distributed in-memory database according to the HASH value of this piece of data. If the same HASH value exists in the distributed in-memory database, filter this piece of data. If the same HASH value does not exist in the distributed in-memory database, store this piece of data in the distributed in-memory database.

[0068] In a specific embodiment, in S4, if the same HASH value does not exist in the distributed in-memory database, store this piece of data in the distributed in-memory database, and load this piece of data into memory and assign a cache expiration time. When all data has passed through the container filtering, we obtain the data values that need to be sampled.

[0069] Based on the four characteristics of the HASH value: 1. The input can be of any length, and the output is of a fixed length. 2. The calculation of the HASH value is relatively fast. 3. Anti-collision property, i.e., uniqueness. 4. Hiding property, also called one-way property. Based on these characteristics, it can be quickly calculated and evenly distributed among the nodes of the distributed in-memory database, which can greatly improve efficiency.

[0070] The present invention uses a distributed memory database as a filtering container, and the data filtering rule is a filtering condition. The filtering container attributes include a distributed cluster server, a data cache size, and a data caching strategy. The filtering condition includes calculating a 128-bit HASH value based on a rule using the MD5 algorithm and constructing a data storage memory database organization rule based on the HASH value, so as to quickly and accurately extract data according to a custom rule in the face of a large amount of big data. The present invention mainly uses a distributed memory database as a container to quickly filter and extract the result information of a large amount of data, which can meet the data sampling effect of various PB levels, and can meet the requirement of obtaining the required sampled result data in a short time without affecting the business efficiency during the business use process. Through the sampling analysis of a large amount of data in various industries, the situation can be quickly grasped, and various early warnings and judgments can be made in advance, which has great significance in the actual combat of various fields such as life production, the development of things, and disaster prediction.

[0071] Further referring to Figure 5 , as an implementation of the above method, the present application provides an embodiment of a data sampling device based on a distributed memory database. This device embodiment corresponds to Figure 1 the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0072] Referring to Figure 5 , a data sampling device based on a distributed memory database includes:

[0073] A HASH value calculation module 101, configured to calculate the HASH value of the current data based on a preset rule when waiting for data to arrive based on a streaming processing program;

[0074] A storage filtering container construction module 102, configured to deploy a database cluster by using 80% of the resources of the system available resource pool for the distributed memory database and divide it into multiple sub-nodes;

[0075] A HASH value storage module 103, configured to write the HASH value of the current data into the distributed memory database;

[0076] A data filtering module 104, when the next data arrives, calculates the HASH value of this data based on a preset rule, and matches according to the HASH value of this data in the distributed memory database. If the same HASH value exists in the distributed memory database, this data is filtered; if the same HASH value does not exist in the distributed memory database, this data is stored in the distributed memory database.

[0077] In a further embodiment, the step of calculating the HASH value of the current data based on a preset rule when waiting for data to arrive based on a streaming processing program specifically includes:

[0078] When the program based on streaming processing waits for data to arrive, it calculates the file size of the current record and differentiates the data greater than 10240KB and less than or equal to 10240KB.

[0079] For the data greater than 10240KB, the first 1024KB and the last 1024KB of the file are intercepted for MD5 calculation. The data less than or equal to 10240KB is converted into byte type and then the full data is reversed for calculation, and then the MD5 full data calculation is performed to calculate the HASH value of the current data.

[0080] In a further embodiment, after the HASH value storage module writes the HASH value of the current data into the distributed memory database, it assigns a cache invalidation time.

[0081] In a further embodiment, in the data filtering module, if the same HASH value does not exist in the distributed memory database, this piece of data is stored in the distributed memory database, and this piece of data is loaded into the memory and assigned a cache invalidation time.

[0082] Next, refer to Figure 6 , which shows a schematic structural diagram of a computer system 200 of an electronic device suitable for implementing the embodiments of the present application. Figure 6 The illustrated electronic device is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0083] As Figure 6 shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 202 or the program loaded from the storage section 208 into the random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the system 200 are also stored. The CPU 201, ROM 202, and RAM 203 are connected to each other through a bus 204. The input / output (I / O) interface 205 is also connected to the bus 204.

[0084] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including, for example, a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. The drive 220 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 220 as needed so that a computer program read therefrom is installed into the storage section 208 as needed.

[0085] Specifically, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 209, and / or installed from the removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the above functions defined in the method of the present application are performed.

[0086] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist separately without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to execute when realizing as Figure 1 shown in.

[0087] It should be noted that the computer-readable storage medium described in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. And in this application, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable storage medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0088] The computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages - such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, alternatively, can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0090] The specific embodiments of the present application have been described above, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all such changes or substitutions should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0091] In the description of the present application, it should be understood that the orientation or positional relationship indicated by terms such as "upper", "lower", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present application. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" before an element does not exclude the presence of a plurality of such elements. The simple fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A data sampling method based on a distributed memory database, characterized in that: The method includes the following steps: S1: When the program based on stream processing waits for data to arrive, calculate the file size of the current record, distinguish data greater than 10240KB and data less than or equal to 10240KB. For data greater than 10240KB, calculate the MD5 of the first 1024KB and the last 1024KB of the intercepted file. For data less than or equal to 10240KB, convert it to byte type and perform reverse calculation on all data, and then perform MD5 calculation on all data, so as to calculate the HASH value of the current data; S2: Build a storage filtering container: Deploy a database cluster of the distributed memory database according to 80% of the system available resource pool, and divide it into multiple sub-nodes; S3: Write the HASH value of the current data into the distributed memory database and assign a cache expiration time; S4: When the next data arrives, calculate the HASH value of this data based on a preset rule, and match it in the distributed memory database according to the HASH value of this data. If the same HASH value exists in the distributed memory database, filter this data. If the same HASH value does not exist in the distributed memory database, store this data in the distributed memory database, and load this data into memory and assign a cache expiration time.

2. A data sampling device based on a distributed memory database, characterized in that: The device includes: A HASH value calculation module, configured to calculate the file size of the current record when the program based on stream processing waits for data to arrive, and distinguish data greater than 10240KB and data less than or equal to 10240KB; for data greater than 10240KB, calculate the MD5 of the first 1024KB and the last 1024KB of the intercepted file, for data less than or equal to 10240KB, convert it to byte type and perform reverse calculation on all data, and then perform MD5 calculation on all data, so as to calculate the HASH value of the current data; A storage filtering container building module, configured to deploy a database cluster of the distributed memory database according to 80% of the system available resource pool, and divide it into multiple sub-nodes; A HASH value storage module, configured to write the HASH value of the current data into the distributed memory database and assign a cache expiration time; A data filtering module, when the next data arrives, calculate the HASH value of this data based on a preset rule, and match it in the distributed memory database according to the HASH value of this data. If the same HASH value exists in the distributed memory database, filter this data. If the same HASH value does not exist in the distributed memory database, store this data in the distributed memory database, and load this data into memory and assign a cache expiration time.

3. An electronic device, including: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in claim 1.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program, when executed by the processor, implements the method described in claim 1.

Images