A big data processing method and system

Abstract

The application provides a big data processing method and system, which can process big data in real time to realize online analysis of big data. The method comprises the following steps: dividing target storage data according to attributes of the target storage data, and obtaining at least one field, the attributes comprising one or more of natural attributes of a user, an address of the user, and part of fields of operation data; determining a key-value code value pair corresponding to a first field in the at least one field according to an attribute of the first field, the key-value code value pair comprising a key value and a code value, the key value being an integer data type, and the code value being the first field or a storage position of the first field; converting the key value and the code value corresponding to the first field into binary data, and storing the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in a storage space in units of bits, wherein the binary data corresponding to fields with the same attribute occupy the same length of the storage space.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a big data processing method and system. Background Technology

[0002] With the continuous development of technologies such as the Internet, the Internet of Things, 5G, artificial intelligence, and cloud computing, more and more data is being generated on the Internet. Because business operations require sophisticated processing, big data processing, data analysis, and digital marketing have become key areas of focus for every Internet company, based on the various data generated on the Internet, and the demand for real-time processing of big data is increasing.

[0003] For example, when a user is using an application (app), they may be directed to a shopping app via pop-ups or by directly opening the app to browse products. The shopping app needs to acquire the user's activity data within the app, process and analyze this data in real time (also known as user behavior) to customize operational strategies and recommend products that the user might be interested in. The result of processing and analyzing this user activity data can be called a user profile. Since there are tens of thousands of users using shopping apps, the volume of user activity data can reach hundreds of millions. Therefore, real-time processing of big data is crucial for shopping apps.

[0004] However, common online analytical processing (OLAP) databases include Hive and Druid. Hive has low computational efficiency, with query latency typically exceeding minutes, making it unsuitable for real-time analysis scenarios. Druid does not support multi-table joins, has low update efficiency, and cannot update user profiles in real time.

[0005] Therefore, how to process big data in real time to achieve online analysis of big data has become an urgent problem to be solved. Summary of the Invention

[0006] This application provides a big data processing method and system that can process big data in real time to achieve online analysis of big data.

[0007] Firstly, a big data processing method is provided, including:

[0008] The target storage data is divided according to the attributes of the target storage data, and at least one field is obtained. The attributes include one or more of the user's natural attributes, the user's address, and some fields of the operation data.

[0009] Based on the attribute of the first field in at least one field, determine the key-value pair corresponding to the first field. The key-value pair includes a key value and a code value. The key value is an integer data type, and the code value is the first field or the location where the first field is stored.

[0010] The key value and code value corresponding to the first field are converted into binary data, and the binary data corresponding to the key value and the binary data corresponding to the code value are stored adjacently in the storage space in bits. Among them, the binary data corresponding to fields with the same attribute occupy the same length of storage space.

[0011] In one example, based on the attributes of the first field in at least one field, the key-value pair corresponding to the first field is determined, including:

[0012] Based on the attribute and first association relationship of the first field in at least one field, determine the key value corresponding to the first field. Use the first field as a code value and combine it with the corresponding key value to obtain the key value code value pair corresponding to the first field. The first association relationship is used to associate the encoded field, attribute, and at least one key value. The at least one key value includes the key value corresponding to the first field. The encoded field includes the first field, or...

[0013] The key value corresponding to the first field is calculated using a 64-bit string hash algorithm based on city hash. The first field is stored on disk, and the location where the first field is stored is used as the code value to determine the key value code value pair corresponding to the first field.

[0014] In one example, storing the first field to disk includes:

[0015] The binary data of the first field is divided into at least one part according to the amount of data that a data page can store, and the data page is stored in the system partition in memory for caching data;

[0016] Store at least one copy of the binary data of the first field in at least one data page;

[0017] During the periodic process of storing data pages from the system partition used for caching data in memory to disk, in the next period, the data page containing the first field is stored to disk, and the corresponding data page from the system partition used for caching data in memory is deleted.

[0018] In one example, based on the attributes of the first field in at least one field, the key-value pair corresponding to the first field is determined, including:

[0019] Based on the stored key-value pairs and at least one field, determine whether the stored fields include the first field;

[0020] If included, determine the key value pair corresponding to the first field based on the stored key value pairs;

[0021] If not included, determine the key value pair corresponding to the first field based on the attribute of the first field in at least one field.

[0022] In one example, before converting the key value and code value corresponding to the first field into binary data, and storing the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in the storage space in bits, the method includes:

[0023] Based on the key value corresponding to the first field, determine the location of the hash block storing the key value corresponding to the first field. The hash block is used to store multiple hash data blocks, and each hash data block is used to store multiple key value code-value pairs. The multiple hash data blocks stored in each hash block adopt a chain structure.

[0024] Based on the location of the hash block storing the key value corresponding to the first field, determine the target location for storing the key value and code value corresponding to the first field.

[0025] In one example, the binary data corresponding to the key value and the binary data corresponding to the code value are stored adjacently in the storage space, in bits, including:

[0026] Based on the target location, and using bits as the unit, the binary data corresponding to the key value and the binary data corresponding to the code value are stored in the storage space.

[0027] In one example, the binary data corresponding to the key value and the binary data corresponding to the code value are stored adjacently in the storage space, including:

[0028] Based on the storage attribute of the first field and the second association relationship, determine the first system partition storing the first field. The second association relationship is used to associate the storage attribute and at least one system partition. The at least one system partition includes the first system partition. The storage attribute includes one or more of the following: the time of data storage, the region where the data was generated, some fields of the data being operated on, the user's natural attributes, and the address.

[0029] The binary data corresponding to the key value and the binary data corresponding to the code value are stored adjacently in the first system partition, which is located in the storage space.

[0030] In one example, the method also includes:

[0031] Obtain a query instruction, which includes at least one conditional data, wherein the conditional data includes the natural attributes, address, or partial fields of the operation data of the user to whom the query instruction is targeted, and the query instruction is used to obtain the data to be queried;

[0032] The target system partition is determined based on at least one conditional data and a second correlation.

[0033] Based on the attributes of each condition data in at least one condition data set, determine the key value corresponding to each condition data set;

[0034] Obtain at least one key value to be queried corresponding to the binary data stored in the target system partition;

[0035] The data to be queried is determined based on at least one key value to be queried and the key value corresponding to each condition data.

[0036] In one example, before determining the data to be queried based on at least one key value to be queried and the key value corresponding to each condition data, the method also includes:

[0037] The update instruction is retrieved and written to memory. The update instruction specifies new data and update conditions, including user attributes, address, or partial fields of the data being manipulated. Specifically, the data to be queried is determined based on at least one key value to be queried and the key value corresponding to each condition data, including:

[0038] Based on the attributes of the new data and the update conditions, determine the key values ​​corresponding to the new data and the key values ​​corresponding to the update conditions;

[0039] Based on the key value corresponding to the update condition and at least one key value to be queried, determine whether at least one key value to be queried needs to be updated;

[0040] If an update is required, update at least one key value to be queried based on the key value corresponding to the new data. Based on the key value corresponding to each condition data and the updated key value to be queried, obtain the key value corresponding to the data to be queried. Determine the data to be queried based on the key value corresponding to the data to be queried.

[0041] If no update is required, obtain the key value corresponding to the data to be queried based on the key value corresponding to each condition data and at least one key value to be queried, and determine the data to be queried based on the key value corresponding to the data to be queried.

[0042] Secondly, a big data processing system is provided, including:

[0043] The field partitioning module is used to partition the target stored data according to the attributes of the target stored data and obtain at least one field. The attributes include one or more of the user's natural attributes, the user's address, and some fields of the operation data.

[0044] The fingerprint encoding module is used to determine the key value code value pair corresponding to the first field based on the attribute of the first field in at least one field. The key value code value pair includes a key value and a code value. The key value is an integer data type, and the code value is the first field or the location where the first field is stored.

[0045] The fingerprint encoding module is also used to convert the key value and code value corresponding to the first field into binary data, and store the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in the storage space in bits. Among them, the binary data corresponding to fields with the same attributes occupy the same length of storage space.

[0046] In this embodiment, the target stored data is divided according to its attributes to obtain at least one field. Then, based on the attributes of the first field among the at least one field, a key value code-value pair corresponding to the first field is determined. The key value code-value pair includes a key value and a code value, where the key value is an integer data type and the code value is the target stored data or the location where the target stored data is stored. Since Chinese characters occupy a large amount of storage space and cannot be directly stored in memory, the fast data processing speed of memory cannot be utilized to process large amounts of data. However, by encoding Chinese characters and obtaining key values ​​as integer data types, the required storage space can be greatly reduced, enabling the storage of large amounts of data in memory. Furthermore, when querying or updating the stored data, the processing speed using key values ​​as the processing object in memory is far greater than the processing speed using Chinese characters as the processing object in current big data processing solutions, facilitating real-time online processing of large amounts of data. Further, this application considers that the operating system must allocate memory in multiples of bytes, resulting in wasted storage space for data whose size is not a multiple of bytes. However, the smallest storage unit in a computer is a bit, and one byte = 8 bits. To improve storage space utilization, the binary data converted from the key value and code value corresponding to the first field are stored adjacently in the storage space, in bits. This means that by allocating memory for the binary data corresponding to multiple columns of data as a whole, the storage space wasted by allocating memory separately for each column in bytes is avoided. This allows the key values ​​of the data to be stored in a smaller memory space, fully utilizing the high speed of memory data processing and further improving the processing speed of large amounts of data. Attached Figure Description

[0047] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic flowchart illustrating a big data processing method provided by this application according to some embodiments;

[0049] Figure 2 This is a schematic diagram of a data storage structure provided by way of example according to some embodiments of this application;

[0050] Figure 3 This is an example of a data storage schematic flowchart provided by this application according to some embodiments;

[0051] Figure 4 This is yet another example of a data storage structure provided by this application based on some embodiments;

[0052] Figure 5 This is a schematic flowchart illustrating yet another example of a big data processing method provided by this application based on some embodiments;

[0053] Figure 6 This is another illustrative flowchart of a big data processing method provided by this application according to some exemplary embodiments;

[0054] Figure 7 This is another illustrative flowchart of a big data processing method provided by this application based on some exemplary embodiments. Detailed Implementation

[0055] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0056] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0057] To improve the real-time performance of big data processing and enable online big data analysis, Figure 1 This is a schematic flowchart illustrating a big data processing method exemplified by some embodiments of this application. The following is a description of the method in conjunction with... Figure 1 This section introduces methods for big data processing, including:

[0058] S110, divide the target storage data according to the attributes of the target storage data and obtain at least one field.

[0059] This attribute includes one or more of the user's natural attributes, the user's address, and certain fields of the operation data.

[0060] For example, a user's natural attributes include one or more attributes such as the user's gender, age, name, and ID number.

[0061] It should be understood that user action data refers to user behavior data on a target application. The target application is the application that generates big data, such as an online shopping platform. User browsing and purchasing behavior data on that platform are collectively referred to as action data. Some fields of action data can be continuous or discontinuous. For example, a continuous field might be "purchase record of product A," while a discontinuous field might be "purchase record" and "product A" used together as query conditions.

[0062] In one example, the target storage data is first obtained, and then the target storage data is divided according to its attributes to obtain at least one field.

[0063] For example, if the target stored data is "Xiaohong, female, Guangzhou, 21 years old", the attributes of the target stored data include name, gender, address, and age. At least one field based on the above attributes includes: "Xiaohong", "female", "Guangzhou", and "21".

[0064] S120, determine the key value pair corresponding to the first field based on the attribute of the first field in at least one field.

[0065] The key-value pair includes a key value and a code value. The key value is an integer data type, and the code value is the first field or the location where the first field is stored.

[0066] In one example, the key-value pairs are stored as follows: Figure 2As shown: First, multiple hash blocks are set up, each corresponding to a hash value. Each hash block stores multiple hash data blocks, which are distributed across memory and disk. Each hash data block stores multiple key-value pairs, and these hash data blocks are linked together in a chained structure. For example, the chained structure... Figure 2 As shown, multiple hash data blocks with the same hash value are connected using next (a pointer to a linked structure).

[0067] In one example, based on the attributes of the first field in the at least one field, the key value code pair corresponding to the first field is determined, including the following two methods. It should be understood that Method 1 and Method 2 are encoding methods for the first field (which can be called fingerprint encoding), and the key value in the key value code pair can be called the encoded value:

[0068] Method 1:

[0069] Based on the attribute and first association relationship of the first field in the at least one field, the key value corresponding to the first field is determined, and the first field is used as a code value and combined with the corresponding key value to obtain the key value code value pair corresponding to the first field. The first association relationship is used to associate the encoded field, the attribute and at least one key value, and the at least one key value includes the key value corresponding to the first field. The encoded field includes the first field.

[0070] The first association relationship is shown in Table 1:

[0071] Table 1

[0072]

[0073] Table 1 is merely an illustrative example and is not intended to limit the scope of this application. The correspondence between other attributes and key values ​​of the first association relationship can be set as needed with reference to Table 1, and will not be elaborated upon here.

[0074] As shown in Table 1, a user's natural attributes correspond to one key value, and multiple users' natural attributes (e.g., age + gender) can also correspond to one key value. Similarly, one or more of a user's natural attributes combined with that user's address can also correspond to one key value. Furthermore, since the key values ​​for fields of different attributes are maintained separately, fields of different attributes can have the same key value (e.g., the first encoded value for gender "male" is 0, and the first encoded value for age "0" is also 0), but fields of the same attribute will have different first encoded values.

[0075] The above method determines the key value corresponding to the encoded field based on the first association relationship. This method is simple, fast, and suitable for encoded fields with a small number of fields or a small amount of data, such as gender (male, female), age (1-150), and address (34 provincial-level administrative regions, each governing a fixed number of prefecture-level cities, and so on). Since the data volume of the encoded fields for these attributes is fixed, they occupy relatively little storage space. Therefore, the key value pairs corresponding to the encoded fields can be stored in memory. Furthermore, because memory processes data quickly, real-time online processing of the encoded fields for these attributes is possible.

[0076] Method 2:

[0077] The key value corresponding to the first field is calculated using the CityHash64 algorithm, the first field is stored on disk, and the location where the first field is stored is used as the code value to determine the key value code value pair corresponding to the first field.

[0078] For user names and ID numbers, which involve hundreds of millions of records, the mapping method in Method 1 is not suitable for calculating key values. Therefore, using the CityHash64 algorithm to calculate key values ​​is appropriate for storing such massive amounts of data. Similarly, the CityHash64 algorithm is also suitable for calculating key values ​​for a large number of fields in the data being manipulated.

[0079] As can be seen from the above embodiments, the encoding method provided by this application can transform large amounts of raw data into smaller amounts of key values, and perform other data processing such as data querying based on the transformed key values, which can improve the speed of big data processing and facilitate real-time online processing of big data.

[0080] In one example, storing the first field to disk includes the following methods:

[0081] The binary data of the first field is divided into at least one part according to the amount of data that a data page can store, and the data page is stored in a system partition in memory for caching data;

[0082] Store at least one copy of the binary data of the first field in at least one of the data pages;

[0083] During the periodic process of storing data pages of the system partition used for caching data in memory to disk, in the next period, the data page containing the first field is stored to disk, and the corresponding data page in the system partition used for caching data in memory is deleted.

[0084] This application considers that the performance of big data processing methods is related to the input / output (IO) of data in memory or on disk. Random IO data reading involves an addressing process, which is relatively inefficient. In current big data processing solutions, each insertion of the encoded field (i.e., the first field, also known as the encoded data) requires memory input, memory output, and disk input. Each IO operation requires random addressing, consuming computational resources. Especially when inserting small amounts of data multiple times, frequent IO operations reduce the overall efficiency of big data insertion. This application divides the binary data converted from the encoded field into multiple parts, with each part stored in a data page. These data pages are then stored in the memory cache according to predefined storage locations. This ensures that all IO operations on the same data page are completed in memory. Subsequently, each data page is periodically written to disk in a single, sorted manner according to its page number and to the predefined storage location, avoiding the waste of computational resources caused by multiple random IO operations.

[0085] In one example, if the total number of cached data pages in memory reaches the memory limit, older data pages are released according to either the first-in, first-out (FIFO) principle or the least-used rule. The least-used rule refers to the rule that the data on a page is processed the fewest times possible.

[0086] For example, before the encoded field is stored on disk, the code value corresponding to the encoded field is stored in the system partition in memory used for caching data. After the encoded field is stored on disk, the code value corresponding to the encoded field is stored in the location of the encoded field on disk.

[0087] In one example, determining the key-value pair corresponding to the first field based on the attribute of the first field in the at least one field includes:

[0088] Based on the stored key value pairs and the first field in the at least one field, determine whether the stored fields include the first field;

[0089] If included, determine the key value pair corresponding to the first field based on the stored key value pair;

[0090] If not included, determine the key value pair corresponding to the first field based on the attribute of the first field among the at least one field.

[0091] Based on the attributes of the first field and the second association relationship, a second system partition is determined to store the key value pairs of the first field. The second system partition is located in memory, which belongs to the storage space.

[0092] Store the key-value pairs of the first field in the second system partition.

[0093] For example, the stored key-value pairs are stored in a specific location in the form of a mapping table.

[0094] In one example, based on the stored key-value pairs and the first field, it is determined whether the stored fields include the current field, including:

[0095] Determine the attributes of the first field;

[0096] If the attribute of the first field is age, gender, or address, determine whether the encoded field of the first association stored in memory includes the first field. Further, if it includes, determine that the stored fields include the first field; if it does not include, determine that the stored fields do not include the first field. The first association also associates the encoded field with a key value. Since the encoded field is a code value, it can be understood that the first association includes multiple key-value code-value pairs.

[0097] If the attribute of the first field is name or a partial field of the data being manipulated, the cityhash64 algorithm is used to calculate the key value of the first field. Then, based on the key value corresponding to the first field, the location of the hash block storing that key value is calculated. Multiple hash data blocks corresponding to that key value are traversed in order from memory to disk to determine whether the key value pairs included in that hash data block contain that key value. Further, if they do, it is determined that the stored fields include the first field; otherwise, it is determined that the stored fields do not include the first field.

[0098] In one example, if the second system partition has no remaining storage space, the key value pairs corresponding to the data in the second system partition that meet the first condition are re-stored to the disk, and the original storage location of the key value pairs corresponding to the data that meet the first condition in memory is used to store the target storage data.

[0099] In the example above, the target data for storage is first acquired. Then, the data is partitioned according to the attributes of the data to be stored, and at least one field is obtained, with different attributes corresponding to different fields. When storing each field, it is first determined whether the currently stored field includes the current field based on the current field and the previously stored key-value pairs. If it does, it means that the current field does not need to be encoded, and fields with the same content as the current field have already been stored. Therefore, the key-value pairs of the current field are directly stored in the second system partition, avoiding the problem of the current field overlapping with previously stored fields, thus preventing serious waste of storage space and improving the performance of the system applying this big data processing method. The second system partition is determined based on the attributes of the current field and the second association relationship, facilitating unified data management later. This system partition resides in memory. Since the data to be stored is hot data, with a high probability of being processed again soon, the encoded values ​​(including key values ​​and code values) corresponding to the data to be stored are preferentially stored in memory. If not, the current field is encoded, and the key-value pairs of the current field are stored. The above method deduplicates the data before storage, determining whether it has been stored before. If it has, only the key-value pairs are stored, without needing to store the original data again. If it hasn't been stored, the data is processed according to the normal procedure. This method avoids wasting storage space and reduces the number of times duplicate data is encoded. For example, consider the data to be stored as "Zhang San, male, browsing product A," which consists of three fields: "Zhang San," "male," and "browsing product A." The already stored fields include "Zhang San," "male," and "browsing product B." Using the above method, only "browsing product A" needs to be encoded and its original content stored. The key-value pairs for the fields "Zhang San" and "male" only need to be copied from the already stored key-value pairs, without needing to be encoded again.

[0100] S130: Convert the key value and code value corresponding to the first field into binary data, and store the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in the storage space in bits.

[0101] Fields with the same attributes occupy the same length of storage space for their corresponding binary data. This storage space includes memory and disk.

[0102] For example, the length of storage space occupied by the key value of the encoded field is related to the attribute of the encoded field.

[0103] For example, the correspondence between the length of storage space occupied by key values ​​and the attributes of the encoded field is shown in Table 1 below:

[0104] Table 2

[0105] Name 64 bits gender 1 bit age 8 bits address 5 bits ID number 8 bits Partial fields of the data being manipulated 32 bits … …

[0106] Table 2 is merely an illustrative example, and this application does not impose any limitations on it; it can be set as needed.

[0107] As exemplified in Table 2, the maximum length of storage space occupied by a key value is 64 bits.

[0108] On the other hand, this application considers that the operating system must allocate memory in multiples of bytes, resulting in wasted storage space for data whose size is not a multiple of bytes. However, the smallest storage unit of a computer is a bit, and one byte = 8 bits. To improve the utilization of storage space, this application provides a bit storage compression method, which allocates memory for the binary data corresponding to multiple columns of data as a whole, avoiding the need to allocate memory for each column separately in bytes. The multiple columns of data are stored in bits, and the binary data of the target column is extracted and then processed.

[0109] For example Figure 3 There are 6 columns of data. Each column requires 8 bytes of memory, and the entire set requires 5 bytes. The binary bit widths of these 6 columns are 4 bits, 9 bits, 12 bits, 6 bits, 4 bits, and 5 bits, respectively. The original data corresponding to each column of binary data may be the user's gender, age, name, ID number, address, or user operation data.

[0110] During storage, the bits of all columns are added together as a whole and a byte array memory is allocated first, and the start and end position information of the binary data in each column is maintained.

[0111] During restoration, taking 12-bit binary data as an example, based on the start and end position information of the 12-bit binary data, the binary data in all the byte arrays occupied by the 12-bit binary data is extracted and converted into long type data. Then, the binary data converted into long type data is right-shifted by 7 bits. Finally, the result after right-shifting by 7 bits is ANDed with the data containing 12 bits of "1" to obtain the original 12-bit binary data. In the data containing 12 bits of "1", the last 12 bits are "1" and the remaining bits are "0".

[0112] Current operating systems have a 64-bit computing performance, meaning that the data length for a single operation is 64 bits. In current storage systems that use bytes as the unit of storage, names, ages, and addresses are stored in bytes. Calculating "Name = XX", "Age = XX", and "Address = XX" requires three comparison operations. In the example above, because data is stored in bits, with names, ages, and addresses stored in 64-bit space, calculating "Name = XX", "Age = XX", and "Address = XX" only requires one comparison operation. Therefore, the above solution further improves the processing capabilities for large amounts of data.

[0113] Therefore, in the example above, the first field uses both fingerprint encoding and bit storage compression methods to store data, which improves the utilization of storage space and enhances the real-time performance of big data query processing.

[0114] For example Figure 4 As shown, the data "Zhang San", "Male", "20", "Guangzhou City", and the message "I know big data, but big data doesn't know me" would require 67 bytes of storage space without fingerprint encoding and bit-based storage compression. Due to the large data volume, this would necessitate storage on disk. However, when querying or processing this massive amount of data online, current big data processing solutions struggle to support fast querying and real-time processing of the stored data. However, by using fingerprint encoding and bit-based storage compression, the data only requires 11 bytes of storage space, significantly reducing disk and memory usage. This allows the data to be stored in memory, enabling fast querying and real-time processing of the stored data.

[0115] In one example, before converting the key value and code value corresponding to the first field into binary data, and storing the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in the storage space in bits, the process is combined with... Figure 2 The data structures and methods also include:

[0116] Based on the key value corresponding to the first field, determine the location of the hash block storing the key value corresponding to the first field. The hash block is used to store multiple hash data blocks, and each hash data block is used to store multiple key value code-value pairs. The multiple hash data blocks stored in each hash block adopt a chain structure.

[0117] Based on the location of the hash block storing the key value corresponding to the first field, determine the target location for storing the key value and code value corresponding to the first field.

[0118] Furthermore, using bits as the unit, the binary data corresponding to the key value and the binary data corresponding to the code value are stored adjacently in the storage space, including:

[0119] Based on the target location, and in bits, the binary data corresponding to the key value and the binary data corresponding to the code value are stored in the storage space, with the storage locations of the key value and the code value corresponding to the first field being adjacent.

[0120] In one example, the binary data corresponding to the key value and the binary data corresponding to the code value are stored adjacently in the storage space, including:

[0121] Based on the storage attribute of the first field and the second association relationship, determine the first system partition storing the first field. The second association relationship is used to associate the storage attribute and at least one system partition. The at least one system partition includes the first system partition. The storage attribute includes one or more of the following: the time of data storage, the region where the data was generated, some fields of the data being operated on, the user's natural attributes, and the address.

[0122] The binary data corresponding to the key value and the binary data corresponding to the code value are stored adjacently in the first system partition, which is located in the storage space.

[0123] The data storage time can be year, month, day, or hour; the data generation region refers to the geographical location of the device storing the data (e.g., province, city, district), which can be the same as or different from the user's address; the partial fields are the fields used to manipulate the data, and these partial fields are set as needed (e.g., "Purchase Product A", "Favorite Product A", "Browse Product A", etc.). The user's natural attributes and address are illustrated in the example above and will not be repeated here.

[0124] For example, the second association is shown in Table 3 below:

[0125] Table 3

[0126]

[0127]

[0128] Table 3 is merely an illustrative example and is not intended to limit the scope of this application. The second relationship can be flexibly set as needed with reference to Table 3, and will not be elaborated further in this application.

[0129] As shown in Table 3, the partitioning method for combining multiple storage attributes (e.g., gender + age) is as follows: under a partition for one attribute, sub-partitions are created based on the other attributes, with different sub-partitions corresponding to different combined attributes. For example, under the gender-male partition, multiple sub-partitions are created based on age, and under the gender-female partition, multiple sub-partitions are created based on age.

[0130] It should be understood that each partition in the system partition can be either a partition in memory or a partition on disk. Because memory processes data faster, partitions in memory can store frequently accessed data, while partitions on disk can store less frequently accessed data. Frequently accessed data refers to data that has been processed more than a preset threshold within the last M days, where M is a positive integer and the preset threshold can be set as needed. This maximizes the utilization of memory's data processing speed and storage space.

[0131] In one example, to facilitate subsequent data retrieval and other processing for a particular user, key-value pairs of stored data belonging to the same user are stored in the same row of the storage space.

[0132] In this embodiment, the target stored data is divided according to its attributes to obtain at least one field. Then, based on the attributes of the first field among the at least one field, a key value code-value pair corresponding to the first field is determined. The key value code-value pair includes a key value and a code value, where the key value is an integer data type and the code value is the target stored data or the location where the target stored data is stored. Since Chinese characters occupy a large amount of storage space and cannot be directly stored in memory, the fast data processing speed of memory cannot be utilized to process large amounts of data. However, by encoding Chinese characters and obtaining key values ​​as integer data types, the required storage space can be greatly reduced, enabling the storage of large amounts of data in memory. Furthermore, when querying or updating the stored data, the processing speed using key values ​​as the processing object in memory is far greater than the processing speed using Chinese characters as the processing object in current big data processing solutions, enabling real-time online processing of large amounts of data. Further, this application considers that the operating system must allocate memory in multiples of bytes, resulting in data whose size is not a multiple of bytes wasting some storage space. However, the smallest storage unit in a computer is a bit, and one byte = 8 bits. To improve storage space utilization, the binary data converted from the key value and code value corresponding to the first field are stored adjacently in the storage space, in bits. This avoids the storage waste caused by allocating memory for each column separately in bytes, by treating the binary data corresponding to multiple columns as a whole. This allows the key values ​​of the data to be stored in a smaller amount of memory.

[0133] Based on the above-mentioned big data storage methods S110-S130, this application also provides a big data query method, including:

[0134] S210, Obtain a query instruction. The query instruction includes at least one condition data, wherein the condition data includes the natural attributes, address, or partial fields of the operation data of the user to whom the query instruction is applied. The query instruction is used to obtain the data to be queried.

[0135] In one example, if the data to be queried includes text, the query instruction is also used to instruct the user to display the text; if the data to be queried includes numerical values, the query instruction is also used to instruct the numerical values ​​to be mathematically calculated.

[0136] For example, if the target user is "Zhang San", at least one conditional data includes: conditional data A is "Name - Zhang San", conditional data B is "Gender - Male", conditional data C is "Address - XX Province XX City", and conditional data D is "Transaction Amount".

[0137] Mathematical calculations can include operations such as summation, averaging, finding the maximum value, and finding the minimum value. For example, summing all transaction amounts.

[0138] It should be understood that in practical applications, query commands may also include only some indications without conditional data, such as only including an indication of the total amount of data to be queried or an indication of the number of users to be queried.

[0139] S220, determine the target system partition based on at least one conditional data and a second association relationship.

[0140] S230, determine the key value corresponding to each condition data based on the attributes of each condition data in at least one condition data.

[0141] For details on how to determine the corresponding key value based on the attributes of the conditional data, please refer to the fingerprint encoding method in S110-S130, which will not be repeated here.

[0142] In the example above, calculating the key values ​​of the conditional data using the fingerprint encoding method significantly reduces the amount of conditional data, enabling subsequent queries of target data within a large dataset using these key values. Compared to directly querying target data using the conditional data, the former is significantly faster, enhancing the real-time performance of big data processing methods. If the data to be queried is encoded using the method described above, storing the data and its encoded key values ​​in memory, the faster data processing speed of memory compared to disk will further improve the speed of data retrieval.

[0143] S240, Obtain at least one key value to be queried corresponding to the binary data stored in the target system partition.

[0144] S250, determine the data to be queried based on at least one key value to be queried and the key value corresponding to each condition data.

[0145] In one example, based on at least one key value to be queried and the key value corresponding to each condition data, the key value corresponding to the data to be queried is determined, and the data to be queried is determined based on the key value corresponding to the data to be queried.

[0146] For example, the target system partition stores at least one query key value row by row, with each row corresponding to one user. Based on the at least one query key value and the key value corresponding to each condition data, the key values ​​of the data to be queried are determined, including:

[0147] Traverse the target system partitions in row-by-row order for the query key values. During the traversal of each row:

[0148] Determine whether the current row contains the key value corresponding to each condition. If it does, use the current row's key value as the key value of the data to be queried, and the traversal process ends; otherwise, continue to the next traversal.

[0149] In one example of this application, when Method 1 is used to encode the field, both the field to be encoded and its key value reside in memory. However, after a period of time, due to updates to hot data in memory, if the field to be encoded is no longer considered hot data, both the field to be encoded and its key value may be re-stored on disk. When Method 2 is used to encode the field, the field to be encoded is stored on disk.

[0150] In one example, to retrieve the key values ​​of the data to be queried, the data to be retrieved includes:

[0151] If the attributes of the data to be queried are the user's age, gender, and address, the key value of the data to be queried is used as the target key value. The location of the hash block storing the target key value is calculated. The hash block is used to store multiple hash data blocks. Each hash data block is used to store multiple key value code-value pairs. The multiple hash data blocks stored in each hash block adopt a chain structure.

[0152] The system iterates through multiple hash data blocks corresponding to the target key value in order from memory to disk. During the iteration of each hash data block:

[0153] The target key value is compared with the key values ​​stored in the current hash data block in the order of the storage locations of all key values ​​in the current hash data block.

[0154] If the current hash data block stores a key value that is the same as the target key value, obtain the target code value corresponding to the target key value. The target code value is the data to be queried, and the traversal process ends.

[0155] If the current hash data block does not store a key value that is the same as the target key value, traverse the next hash data block.

[0156] In the above example, if the attributes of the data to be queried are the user's age, gender, and address, then the data to be queried is stored as a code value in the storage space. Therefore, after calculating the location of the hash block storing the target key value, the target key value code value pair can be queried from the key value code value pairs in that hash block.

[0157] In one example, as Figure 2 Under the data structure shown, the data to be queried is obtained based on the key value of the data to be queried, including:

[0158] If the attribute of the data to be queried is a name or a partial field of the data to be operated, the key value of the data to be queried is used as the target key value, and the location of the hash block storing the target key value is calculated. The hash block is used to store multiple hash data blocks, each hash data block is used to store multiple key value code-value pairs, and the multiple hash data blocks stored in each hash block adopt a chain structure.

[0159] The system iterates through multiple hash data blocks corresponding to the target key value in order from memory to disk. During the iteration of each hash data block:

[0160] Compare the target key value with the key values ​​stored in the current hash data block in the order of their storage locations.

[0161] If the current hash data block stores a key value that is the same as the target key value, obtain the target code value corresponding to the target key value, determine the target storage location based on the target code value, obtain the data to be queried based on the target storage location, and the traversal process ends;

[0162] If the current hash data block does not store the same key value as the target key value, traverse the next hash data block.

[0163] For example Figure 2As shown, the key value "1236458795" of the data to be queried, "original long text data," is used as the target key value. The calculation `key%N = 1` is performed, where N is an integer set as needed. The result "1" indicates that the target key value is stored in a hash block with a hash value of 1. Then, the location of the hash block is determined based on the hash value `hash = 1`. First, the key values ​​stored in the hash data block with a hash value of 1 are traversed in memory. This hash data block sequentially stores "next, key, value, key, value, key, value". If the current hash data block stores a key value identical to the target key value, the corresponding target code value `value` is obtained, and the target storage location is determined based on this target code value. This traversal process then ends. If the hash data block in memory does not store a key value that is the same as the target key value, search for the next hash data block stored on disk according to next, and continue to compare the key value stored in the hash data block with the target key value. If the current hash data block stores a key value that is the same as the target key value, obtain the target code value corresponding to the target key value, and determine the target storage location according to the target code value value. The traversal process ends. Otherwise, traverse the next hash data block.

[0164] In the example above, the key value of the data to be queried is used as the key value to calculate the location of the hash block. This greatly narrows down the range of data to be searched. Furthermore, following the order from memory to disk, the key values ​​stored in the hash data block corresponding to the hash block are traversed. Since hot data is stored in memory, if the data to be queried is hot data, the corresponding key value is more likely to be stored in memory. Because memory processes data faster, this further improves the efficiency of data retrieval compared to the current solution of directly searching for data on the disk. Even if the key value corresponding to the data to be queried is not stored in memory as hot data, according to the chain structure, searching for the key value on the disk is still faster than the current solution of searching for the original data on the disk because the amount of key value data is greatly reduced compared to the original data before encoding. (It should be noted that because the amount of original data is large, current solutions for processing large amounts of data cannot store the original data in memory and can only store the original data on the disk. Therefore, the processing of large amounts of data can only be carried out on the disk, not in memory, and due to the large amount of data and the limitations of disk processing speed, the current solution is slow in terms of data retrieval speed.) Furthermore, if the data to be queried has attributes such as name, or involves only certain fields, resulting in a large volume of data, then the data to be queried will be encoded using the first encoding method in Method Two. The code value corresponding to the data to be queried is stored on the disk. Therefore, it is necessary to determine the target storage location based on the target code value before retrieving the queried data from the target storage location.

[0165] It should be understood that obtaining the data to be queried, based on the key values ​​of the data to be retrieved, can be seen as a process of decoding the key values.

[0166] For example, if the retrieved data is text, the method further includes:

[0167] The retrieved text will be displayed to the user.

[0168] If the queried data is a numerical value and the query instruction is also used to instruct mathematical calculations to be performed on the numerical value, the method further includes:

[0169] Perform mathematical calculations on the retrieved values ​​based on the query command.

[0170] In this embodiment, conditional data is fingerprinted to obtain the key values ​​corresponding to the conditional data. Since the first encoding is used to convert the encoded data into key values ​​of integer data type, the amount of key values ​​in the conditional data is significantly reduced compared to the conditional data before encoding. Therefore, compared to the current method of directly searching for the data to be queried in a large amount of searched data based on the conditional data, this application transforms the retrieval object from conditional data and searched data into their respective key values, then finds the key values ​​corresponding to the data to be queried based on the key values ​​of the conditional data, and then obtains the target storage location of the data to be queried based on the key values ​​of the data to be queried, and extracts the data to be queried from the target storage location, thus greatly improving the speed of data retrieval. On the other hand, since this application stores the binary numbers corresponding to the key values ​​of the stored data (including the data to be queried) in bits, compared to the current storage scheme in bytes, it avoids the problem of wasted storage space when the amount of stored data is not an integer multiple of bytes, and realizes the storage of the key values ​​of the data in a smaller memory space.

[0171] This application considers the dynamic nature of data, meaning that hot data changes over time. In other words, hot data in memory is not static and needs to be automatically updated. As seen in the above embodiments, when data is queried, it is considered current hot data, with a high probability of being queried again soon. If the data to be queried is stored on disk, it is not conducive to future queries. Therefore, this application proposes the following method for automatically updating hot data in memory:

[0172] If the target key value is not stored in the hash data block in memory, determine the key value code pair that meets the first condition stored in the hash data block in memory, and the first storage location of the key value code pair that meets the first condition in memory.

[0173] The target key value and the target code value are located in the second storage location on the disk;

[0174] The key value pair that meets the first condition is stored on the disk of the second storage location;

[0175] The target key value and the target code value are stored in the memory of the first storage location.

[0176] The first condition is that the number of times the data has been processed in the most recent M days is less than a preset threshold. In other words, the data corresponding to the key value pairs that meet the first condition are non-hotspot data.

[0177] In one example, if there is no key-value pair that meets the first condition in the hash data block located in memory, the search continues in the order of the storage locations of each hash block until a key-value pair that meets the first condition is obtained.

[0178] In the example above, by swapping the storage location of the key value pair corresponding to the data to be queried with the storage location of the key value pair in memory that meets the first condition, the hot data in memory is automatically updated, which greatly improves data utilization and algorithm performance.

[0179] In one example, this application also provides a data update method for big data, which further includes the following steps before determining the data to be queried based on at least one key value to be queried and the key value corresponding to each condition data:

[0180] The update instruction is retrieved and written to memory. The update instruction specifies new data and update conditions, including user attributes, address, or partial fields of the data being manipulated. Specifically, the data to be queried is determined based on at least one key value to be queried and the key value corresponding to each condition data, including:

[0181] Based on the attributes of the new data and the update conditions, determine the key values ​​corresponding to the new data and the key values ​​corresponding to the update conditions;

[0182] Based on the key value corresponding to the update condition and at least one key value to be queried, determine whether at least one key value to be queried needs to be updated;

[0183] If an update is required, update at least one key value to be queried based on the key value corresponding to the new data. Based on the key value corresponding to each condition data and the updated key value to be queried, obtain the key value corresponding to the data to be queried. Determine the data to be queried based on the key value corresponding to the data to be queried.

[0184] If no update is required, obtain the key value corresponding to the data to be queried based on the key value corresponding to each condition data and at least one key value to be queried, and determine the data to be queried based on the key value corresponding to the data to be queried.

[0185] For example, the update method further includes: establishing a correspondence between the key values ​​of the new data and the target storage location where the new data is cached in memory.

[0186] For example, determining whether the at least one query key value needs to be updated based on the key value of the update condition and the at least one query key value includes:

[0187] Iterate through the at least one key value to be queried, and during each iteration:

[0188] Compare the current query key value with the key value corresponding to the update condition. If they are the same, change the current query key value to the key value corresponding to the new data, and the traversal process ends; otherwise, determine that the current query key value does not need to be updated.

[0189] For example, the update method further includes:

[0190] The binary data converted from the new data is stored in memory as data pages in a partition used for caching data;

[0191] If the storage space of the current data page is full and the binary data converted from the new data has not been completely stored, the remaining binary data will be stored in the next data page until all the binary data converted from the new data is cached in memory.

[0192] During the periodic process of storing cached data in memory to disk, when the next cycle arrives, the data page where the new data is converted into binary data is stored is stored in the disk location where the original data's binary data page is stored, thus overwriting the original data page, and the data page where the new data's binary data is cached in memory is deleted.

[0193] It should be understood that since the storage location of the original data on the disk is maintained when the original data is stored, the data pages where the binary data converted from the new data is stored can overwrite the data pages where the original data is stored.

[0194] In the above example, to avoid the performance impact of frequent disk I / O operations due to frequent data updates, this application caches new data in memory and updates the original data and its key values ​​only when querying the partition where the key values ​​of the original data are stored. This achieves a situation where, during the periodic storage of cached data in memory to disk, when multiple data points are updated on the same data page, all the new data on that page are stored on disk simultaneously. Compared to performing a disk I / O operation for each data update, this method reduces the number of disk I / O operations, fully utilizes memory resources, and improves the speed of data updates.

[0195] The following examples, using the aforementioned embodiments, illustrate data insertion methods applied to big data, such as... Figure 5 As shown, the method includes:

[0196] Step a1: Obtain the data to be inserted, which includes: "Zhang San", "Male", "20", "Address 1", "Browse Product A".

[0197] Step a2: Encode the data to be inserted using fingerprinting to obtain key values: 0, 1, 20, 153, 1349, and maintain the original data (i.e., the storage location of the data to be inserted on the disk). Key value "0" corresponds to the storage location of the code value "Zhang San", key value "1" corresponds to the code value "Male", key value "20" corresponds to the code value 20, key value "153" corresponds to the storage location of the code value "Address 1", and key value "1349" corresponds to the storage location of the code value "Browse Product A".

[0198] Step a3: Based on the address "address 1" in the data to be inserted, calculate the partition used to store the binary data corresponding to the key value as system partition 1.

[0199] Step a4: Convert the key value into binary data and store it in the data page of system partition 1.

[0200] In the example above, the data to be inserted is first fingerprinted to obtain key values ​​of integer type, which greatly reduces the amount of original data and allows the key values ​​to be stored in memory. Taking advantage of the fast data processing speed of memory, it is beneficial to improve the processing speed of the data in the future, and realizes real-time online data processing in big data with hundreds of millions of data.

[0201] The following examples, using the aforementioned embodiments, illustrate data query methods applied to big data, such as... Figure 6 As shown, the method includes:

[0202] Step b1: Obtain the query instruction, which includes the following conditions: name "Zhang San" and address "Address 1".

[0203] Step b2: Encode the conditional data using fingerprints to obtain the key values ​​corresponding to the conditional data: 0, 1.

[0204] Step b3: Based on the conditional data and the second association, calculate the key values ​​of the data to be queried and store them in the system partition 1.

[0205] Step b4: Following the storage location order of the data pages in system partition 1, traverse the binary data stored in each data page. During each traversal:

[0206] Convert binary data to decimal to obtain the key value corresponding to the binary data;

[0207] Based on the magnitudes of the key values ​​corresponding to the current binary data and the key values ​​corresponding to the conditional data, determine whether the key values ​​corresponding to the current binary data contain the key values ​​corresponding to the conditional data.

[0208] If it exists, determine the target storage location of the original data (i.e. the data to be queried) based on the key value corresponding to the binary data, extract all the data of the user stored in the row where the target storage location is located, and display the data to be queried to the user: "Zhang San", "Male", "20", "Address 1", "Browse Product A", and the traversal process ends.

[0209] If it does not exist, proceed to the next iteration.

[0210] In the example above, by storing the key values ​​of the data to be queried in memory, the key values ​​corresponding to the data to be queried can be found in memory based on the key values ​​of the conditional data and the corresponding key values ​​of the binary data in the memory data page, thus leveraging the high speed of memory data processing. Furthermore, since the amount of key values ​​is significantly reduced compared to the original data, the above data query method achieves the effect of online real-time querying of data in large datasets.

[0211] The following examples, using the aforementioned embodiments, illustrate data update methods applied to big data, such as... Figure 7 As shown, the method includes:

[0212] Step c1: Obtain the update command, which includes: update condition "Zhang San, Address 1", new data "Zhang San, Male, 20, Address 1, Browse Product B".

[0213] Step c2: Obtain the query instruction, which includes the condition data: "Zhang San, address 1".

[0214] Step c3: Encode the conditional data using fingerprints to obtain key values ​​0 and 1.

[0215] Step c4: Calculate the partition for storing the data to be queried based on the conditional data, which is system partition 1.

[0216] Step c5: Encode the update conditions and new data with fingerprints to obtain their respective key values, which are 0,1 and 0,1,20,153,1348 respectively.

[0217] Step c6: Following the storage location order of the data pages in system partition 1, traverse each line of binary data (each line of binary data corresponds to one user) of each data page in partition 1. During each traversal:

[0218] Convert the binary data of the current row to decimal; the converted value is the key value to be queried in the current row.

[0219] Determine whether the key value to be queried in the current row includes the key value corresponding to the update condition. If it does, determine that the key value to be queried in the current row needs to be updated; otherwise, determine that the key value to be queried in the current row does not need to be updated.

[0220] If an update is needed, obtain the key value corresponding to the new data, convert the key value into binary data, and update the corresponding data stored in the binary file in system partition 1. Then, based on the key value of the new data and the key value of the conditional data, determine whether the new data is the data to be queried. If so, extract the new data according to the cache location corresponding to the key value of the new data, and display the data to be queried (i.e., the new data) to the user as: "Zhang San", "Male", "20", "Address 1", "Browse Product B", and the traversal process ends. If not, proceed to the next traversal.

[0221] If no update is needed, proceed to the next iteration.

[0222] In the above example, to avoid the performance impact of frequent disk I / O operations due to frequent data updates, this application caches new data in memory and updates the original data and its key value only when querying the partition where the fingerprint code value of the original data is stored. This achieves a unified storage of multiple new data entries on the same data page when multiple data entries are updated on the same data page, during the periodic storage of cached data to disk. Compared to performing a disk I / O operation for each data update, this method reduces the number of disk I / O operations, fully utilizes memory resources, and improves the speed of data updates.

[0223] This application also provides a big data processing system, characterized in that it includes:

[0224] The field partitioning module is used to partition the target stored data according to the attributes of the target stored data and obtain at least one field. The attributes include one or more of the user's natural attributes, the user's address, and some fields of the operation data.

[0225] The fingerprint encoding module is used to determine the key value code value pair corresponding to the first field based on the attribute of the first field in at least one field. The key value code value pair includes a key value and a code value. The key value is an integer data type, and the code value is the first field or the location where the first field is stored.

[0226] The fingerprint encoding module is also used to convert the key value and code value corresponding to the first field into binary data, and store the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in the storage space in bits. Among them, the binary data corresponding to fields with the same attributes occupy the same length of storage space.

[0227] In one example, the fingerprint encoding module also includes:

[0228] The data partitioning submodule is used to divide the binary data of the first field into at least one part according to the amount of data that a data page can store, wherein the data page is stored in a system partition in memory for caching data;

[0229] A cached data submodule is used to store at least one copy of the binary data of the first field in at least one of the data pages;

[0230] The cached data submodule is also used to, during the periodic process of storing data pages of the system partition used for caching data in memory to disk, store the data page containing the first field to disk in the next period and delete the corresponding data page in the system partition used for caching data in memory.

[0231] In one example, the system also includes a deduplication module:

[0232] The deduplication module is used to determine whether the stored fields include the first field based on the stored key-value pairs and the first field in the at least one field;

[0233] If included, the fingerprint encoding module is used to determine the key value code pair corresponding to the first field based on the stored key value code pair;

[0234] If not included, the fingerprint encoding module is used to determine the key value code pair corresponding to the first field based on the attribute of the first field in the at least one field.

[0235] In one example, the fingerprint encoding module also includes:

[0236] The hash block location acquisition submodule is used to determine the location of the hash block storing the key value corresponding to the first field based on the key value corresponding to the first field. The hash block is used to store multiple hash data blocks, each hash data block is used to store multiple key value code-value pairs, and the multiple hash data blocks stored in each hash block adopt a chain structure.

[0237] The target location acquisition submodule is used to determine the target location for storing the key value and code value corresponding to the first field based on the location of the hash block storing the key value corresponding to the first field.

[0238] In one example, the fingerprint module also includes;

[0239] The data partitioning submodule is used to determine the first system partition storing the first field based on the storage attribute of the first field and the second association relationship. The second association relationship is used to associate the storage attribute with at least one system partition. The at least one system partition includes the first system partition. The storage attribute includes one or more of the following: data storage time, data generation region, partial fields of the data being operated on, user's natural attributes, and address.

[0240] The fingerprint module is used to store the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in the first system partition, which is located in the storage space.

[0241] In one example, the system also includes an instruction acquisition module and a query data acquisition module, specifically:

[0242] The instruction acquisition module is used to acquire query instructions. The query instructions include at least one set of condition data, which includes the natural attributes, address, or partial fields of the operation data of the user to whom the query instructions are applied. The query instructions are used to acquire the data to be queried.

[0243] The data partitioning submodule is used to determine the target system partition based on at least one conditional data and a second association relationship;

[0244] The fingerprint encoding module is used to determine the key value corresponding to each condition data based on the attribute of each condition data in at least one condition data.

[0245] The query data acquisition module is used to obtain at least one key value to be queried corresponding to the binary data stored in the target system partition;

[0246] The data retrieval module is also used to determine the data to be queried based on at least one key value to be queried and the key value corresponding to each condition data.

[0247] In one example, the system also includes an update instruction acquisition module, an update judgment module, and a data update module:

[0248] The instruction acquisition module is also used to acquire update instructions and write them into memory. The update instructions indicate new data and update conditions. Update conditions include the user's natural attributes, address, or partial fields of the data being manipulated. Specifically, the data to be queried is determined based on at least one key value to be queried and the key value corresponding to each condition data, including:

[0249] The fingerprint encoding module is also used to determine the key value corresponding to the new data and the key value corresponding to the update condition based on the attributes of the new data and the update condition.

[0250] The update judgment module is used to determine whether at least one key value to be queried needs to be updated based on the key value corresponding to the update condition and at least one key value to be queried.

[0251] If an update is required, the data update module is used to update at least one key value to be queried based on the key value corresponding to the new data. Based on the key value corresponding to each condition data and the updated key value to be queried, the key value corresponding to the data to be queried is obtained, and the data to be queried is determined based on the key value corresponding to the data to be queried.

[0252] If no update is required, the query data acquisition module is used to obtain the key value corresponding to the data to be queried based on the key value corresponding to each condition data and at least one key value to be queried, and to determine the data to be queried based on the key value corresponding to the data to be queried.

[0253] In one example, the above-mentioned big data processing system also includes the following modules (this big data processing system can also be referred to as a database, and in addition to the solutions in the embodiments of this application, it also includes other basic functions of current databases. Functions can be added as needed during implementation, and will not be described in detail in this application):

[0254] The cached query module has expiration time and capacity limits. Structured Query Language (SQL) queries that hit the cache are returned directly, or the execution plan is cached to avoid secondary parsing of the SQL query.

[0255] The SQL parsing module parses SQL into database internal execution classes such as insert, select, and data operators for conditional judgment and data column extraction.

[0256] Other implementations and effects of this device are described in the section on big data processing methods, and will not be repeated here. The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0257] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0258] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0259] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0260] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0261] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A big data processing method, characterized in that, include: The target stored data is divided according to the attributes of the target stored data, and at least one field is obtained. The attributes include one or more of the user's natural attributes, the user's address, and some fields of the operation data. Based on the attributes of the first field in the at least one field, determine the key-value pair corresponding to the first field, wherein the key-value pair includes a key value and a code value, the key value being an integer data type, and the code value being the first field or the location where the first field is stored, wherein determining the key-value pair corresponding to the first field based on the attributes of the first field in the at least one field includes: Based on the attribute and first association relationship of the first field in the at least one field, a key value corresponding to the first field is determined. The first field is used as a code value and combined with the corresponding key value to obtain a key value code value pair corresponding to the first field. The first association relationship is used to associate the encoded field, the attribute, and at least one key value. The at least one key value includes the key value corresponding to the first field. The encoded field includes the first field, or... The CityHash64 algorithm is used to calculate the key value corresponding to the first field, the first field is stored on disk, and the location where the first field is stored is used as the code value to determine the key value code value pair corresponding to the first field. Based on the key value corresponding to the first field, the location of the hash block storing the key value corresponding to the first field is determined, wherein the hash block is used to store multiple hash data blocks, each hash data block is used to store multiple key value code-value pairs, and the multiple hash data blocks stored in each hash block adopt a chain structure; Based on the location of the hash block storing the key value corresponding to the first field, determine the target location for storing the key value and code value corresponding to the first field; The key value and code value corresponding to the first field are converted into binary data, and the binary data corresponding to the key value and the binary data corresponding to the code value are stored adjacently in the storage space, in bits. The binary data corresponding to fields with the same attributes occupy the same length of the storage space. The length of the storage space occupied by the key value of the encoded field is related to the attributes of the encoded field. The step of storing the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in the storage space, in bits, includes: Based on the target location, and in bits, the binary data corresponding to the key value and the binary data corresponding to the code value are stored in the storage space, with the storage locations of the key value and the code value corresponding to the first field being adjacent. Obtain a query instruction, the query instruction including at least one condition data, wherein the condition data includes the natural attributes, address or partial fields of the operation data of the user to whom the query instruction is targeted, and the query instruction is used to obtain the data to be queried; Based on the at least one conditional data and the second association, a target system partition is determined, wherein the second association is used to associate the storage attributes of the associated field with at least one system partition; Based on the attribute of each of the at least one conditional data, determine the key value corresponding to each conditional data; Obtain at least one key value to be queried corresponding to the binary data stored in the target system partition; Based on the at least one query key value and the key value corresponding to each conditional data, the data to be queried is determined. This determination includes: determining the key value corresponding to the data to be queried based on the at least one query key value and the key value corresponding to each conditional data; and determining the data to be queried based on the key value corresponding to the data to be queried. This step includes traversing multiple hash data blocks corresponding to the target key value in order from memory to disk. During this traversal, the method further includes automatically updating hot data in memory. Hot data refers to data processed more than a preset threshold within the last M days, where M is a positive integer.

2. The method according to claim 1, characterized in that, The step of storing the first field to the disk includes: The binary data of the first field is divided into at least one part according to the amount of data that a data page can store, and the data page is stored in a system partition in memory for caching data; Store at least one copy of the binary data of the first field in at least one of the data pages; During the periodic process of storing data pages of the system partition used for caching data in memory to disk, in the next period, the data page containing the first field is stored to disk, and the corresponding data page in the system partition used for caching data in memory is deleted.

3. The method according to claim 1 or 2, characterized in that, The step of determining the key value pair corresponding to the first field based on the attribute of the first field in the at least one field includes: Based on the stored key value pairs and the first field in the at least one field, determine whether the stored fields include the first field; If included, determine the key value pair corresponding to the first field based on the stored key value pair; If not included, determine the key value pair corresponding to the first field based on the attribute of the first field among the at least one field.

4. The method according to claim 1 or 2, characterized in that, The step of storing the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in the storage space includes: Based on the storage attribute of the first field and the second association relationship, a first system partition storing the first field is determined. The second association relationship is used to associate the storage attribute with at least one system partition. The at least one system partition includes the first system partition. The storage attribute includes one or more of the following: data storage time, data generation region, partial fields of the data being operated on, user's natural attributes, and address. The binary data corresponding to the key value and the binary data corresponding to the code value are stored adjacently in the first system partition, which is located in the storage space.

5. The method according to claim 1, characterized in that, Before determining the data to be queried based on the at least one query key value and the key value corresponding to each condition data, the method further includes: An update instruction is obtained and written to memory. This update instruction indicates new data and update conditions, including user attributes, address, or partial fields of the operation data. The data to be queried is determined based on the at least one query key value and the key value corresponding to each condition data, including: Based on the attributes of the new data and the update conditions, determine the key values ​​corresponding to the new data and the key values ​​corresponding to the update conditions; Based on the key value corresponding to the update condition and the at least one key value to be queried, determine whether the at least one key value to be queried needs to be updated; If an update is required, update the at least one query key value according to the key value corresponding to the new data, obtain the key value corresponding to the data to be queried according to the key value corresponding to each condition data and the updated at least one query key value, and determine the data to be queried according to the key value corresponding to the data to be queried. If no update is required, obtain the key value corresponding to the data to be queried based on the key value corresponding to each condition data and the at least one key value to be queried, and determine the data to be queried based on the key value corresponding to the data to be queried.

6. The method according to claim 1 or 2, characterized in that, The process of determining the data to be queried based on the key values ​​corresponding to the data to be queried includes: If the attributes of the data to be queried are the user's age, gender, and address, the key value of the data to be queried is used as the target key value, and the location of the hash block storing the target key value is calculated. The system iterates through multiple hash data blocks corresponding to the target key value in order from memory to disk. During the iteration of each hash data block: The target key value is compared with the key values ​​stored in the current hash data block in the order of the storage locations of all key values ​​in the current hash data block. If the current hash data block stores a key value that is the same as the target key value, obtain the target code value corresponding to the target key value. The target code value is the data to be queried, and the traversal process ends. If the current hash data block does not store a key value that is the same as the target key value, traverse the next hash data block; Alternatively, determining the data to be queried based on the key values ​​corresponding to the data to be queried includes: If the attribute of the data to be queried is a name or a partial field of the data to be operated on, the key value of the data to be queried is used as the target key value, and the location of the hash block storing the target key value is calculated. The system iterates through multiple hash data blocks corresponding to the target key value in order from memory to disk. During the iteration of each hash data block: Compare the target key value with the key values ​​stored in the current hash data block in the order of their storage locations. If the current hash data block stores a key value that is the same as the target key value, obtain the target code value corresponding to the target key value, determine the target storage location based on the target code value, obtain the data to be queried based on the target storage location, and the traversal process ends; If the current hash data block does not store the same key value as the target key value, traverse the next hash data block.

7. The method according to claim 6, characterized in that, The steps for automatically updating hot data in memory include: If the target key value is not stored in the hash data block in memory, determine the key value code pair that meets the first condition stored in the hash data block in memory, and the key value code pair that meets the first condition is located in the first storage location in memory; The target key value and the target code value are located in a second storage location on the disk; The key value pairs that meet the first condition are stored on the disk of the second storage location; The target key value and the target code value are stored in the memory of the first storage location, and the first condition is that the number of times they are processed in the most recent M days is less than the preset threshold.

8. A big data processing system, characterized in that, include: The field partitioning module is used to partition the target stored data according to the attributes of the target stored data and obtain at least one field, wherein the attributes include one or more of the user's natural attributes, the user's address, and some fields of the operation data; A fingerprint encoding module is used to determine a key-value pair corresponding to the first field based on the attributes of the first field in the at least one field. The key-value pair includes a key value and a code value, wherein the key value is an integer data type and the code value is the first field or the location where the first field is stored. The step of determining the key-value pair corresponding to the first field based on the attributes of the first field in the at least one field includes: Based on the attribute and first association relationship of the first field in the at least one field, a key value corresponding to the first field is determined. The first field is used as a code value and combined with the corresponding key value to obtain a key value code value pair corresponding to the first field. The first association relationship is used to associate the encoded field, the attribute, and at least one key value. The at least one key value includes the key value corresponding to the first field. The encoded field includes the first field, or... The CityHash64 algorithm is used to calculate the key value corresponding to the first field, the first field is stored on disk, and the location where the first field is stored is used as the code value to determine the key value code value pair corresponding to the first field. The hash block location acquisition submodule included in the fingerprint encoding module is used to determine the location of the hash block storing the key value corresponding to the first field based on the key value corresponding to the first field. The hash block is used to store multiple hash data blocks, each hash data block is used to store multiple key value code-value pairs, and the multiple hash data blocks stored in each hash block adopt a chain structure. The target location acquisition submodule included in the fingerprint encoding module is used to determine the target location for storing the key value and code value corresponding to the first field based on the location of the hash block storing the key value corresponding to the first field. The fingerprint encoding module is further configured to convert the key value and code value corresponding to the first field into binary data, and store the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in a storage space, in bits. Specifically, the binary data corresponding to fields with the same attributes occupy the same length of the storage space, and the length of the storage space occupied by the key value of the encoded field is related to the attributes of the encoded field. The step of storing the binary data corresponding to the key value and the binary data corresponding to the code value adjacently in the storage space, in bits, includes: Based on the target location, and in bits, the binary data corresponding to the key value and the binary data corresponding to the code value are stored in the storage space, with the storage locations of the key value and the code value corresponding to the first field being adjacent. The instruction acquisition module is used to acquire a query instruction, which includes at least one condition data, wherein the condition data includes the natural attributes, address, or partial fields of the operation data of the user to whom the query instruction is applied, and the query instruction is used to acquire the data to be queried. The data partitioning submodule included in the fingerprint encoding module is used to determine the target system partition based on the at least one conditional data and the second association relationship, wherein the second association relationship is used to associate the storage attributes of the field with at least one system partition; The fingerprint encoding module is further configured to determine the key value corresponding to each condition data according to the attribute of each condition data in the at least one condition data; The query data acquisition module is used to acquire at least one key value to be queried corresponding to the binary data stored in the target system partition; The query data acquisition module is further configured to determine the data to be queried based on the at least one query key value and the key value corresponding to each condition data. The determination of the data to be queried based on the at least one query key value and the key value corresponding to each condition data includes: determining the key value corresponding to the data to be queried based on the at least one query key value and the key value corresponding to each condition data; determining the data to be queried based on the key value corresponding to the data to be queried; and determining the data to be queried based on the key value corresponding to the data to be queried includes the step of traversing multiple hash data blocks corresponding to the target key value in order from memory to disk. The process of traversing multiple hash data blocks corresponding to the target key value in order from memory to disk also includes the step of automatically updating hot data in memory. Hot data refers to data whose processing frequency exceeds a preset threshold within the last M days, where M is a positive integer.

Images