Data table data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data table data processing method and device, electronic equipment and a storage medium. The method comprises the steps that service data to be written into a data table is received, the data table comprises a plurality of data partitions divided according to rows, and each data partition comprises a plurality of data buckets divided according to rows; the business data is written into a to-be-written data sub-bucket in a to-be-written data partition as a line, the to-be-written data sub-bucket is any one of a plurality of data sub-buckets of the to-be-written data partition, each data partition comprises a plurality of column clusters divided according to columns, each column cluster is used for writing feature data of a plurality of associated types, and each column cluster is used for writing the feature data of the associated types. The column clusters are physically and independently stored; a range query request is received, and the range query request comprises a plurality of to-be-queried types; and reading feature data of each row in each column cluster including a plurality of to-be-queried types from at least one data partition. According to the invention, high-efficiency storage and high-performance range scanning requirements can be considered.

Description

Technical Field

[0001] The present application relates to the technical field of data tables, and in particular to a data processing method, device, electronic device and storage medium for a data table. Background Art

[0002] At present, common mass data storage includes Hbase, object storage, ClickHouse, etc. Among them, Hbase is a highly reliable, high-performance, and scalable distributed storage system that supports column cluster storage and retrieval of massive data. The underlying storage adopts a sparse table data model, which takes up very little space for large-scale sparse data storage. However, column clusters are actually wide column storage, and row storage is still used inside column clusters. In actual business applications, column clusters are limited, and there is a serious waste of bandwidth when accessing some columns. On the other hand, Hbase has poor performance in large-scale scanning queries. Object storage is a distributed storage that supports massive data without format restrictions and millisecond-level response, but object storage itself does not support range scanning. Full scanning requires additional storage components to store the key (Object_Key) of the object, which is implemented as a single query, and has low performance. ClickHouse is a column storage data management system for data analysis, which supports online real-time query without any data preprocessing, and realizes millisecond-level analysis at the billion-level data volume. ClickHouse implements pure column storage, which can access features on demand and access bandwidth without waste. However, in terms of application, ClickHouse is more inclined towards real-time analysis, and has poor support for data range scanning and single data update (which will generate a large number of small partition files, making the performance during analysis worse).

[0003] In other words, there is no effective solution in the relevant technology for how to balance the requirements of efficient storage and high-performance range scanning for massive data. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium and computer program product for a data table, which can take into account both efficient storage and high-performance range scanning requirements.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data processing method for a data table, including:

[0007] Receive business data to be written into a data table, wherein the data table includes a plurality of data partitions divided by rows, and each of the data partitions includes a plurality of data buckets divided by rows;

[0008] Writing the business data as a row into a data bucket to be written in the data partition to be written, wherein the data partition to be written is any one of the multiple data partitions, the data bucket to be written is any one of the multiple data buckets included in the data partition to be written, each of the data partitions includes a plurality of column clusters divided by columns, each of the column clusters is used to write characteristic data of multiple associated types, and the column clusters are physically stored independently;

[0009] Receiving a range query request, wherein the range query request includes multiple types to be queried;

[0010] From at least one of the data partitions, characteristic data of each row in each column cluster including the multiple types to be queried is read.

[0011] The present application provides a data processing device for a data table, including:

[0012] A receiving module, configured to receive business data to be written into a data table, wherein the data table includes a plurality of data partitions divided by rows, and each of the data partitions includes a plurality of data buckets divided by rows;

[0013] A writing module, used for writing the business data as a row into a data bucket to be written in the data partition to be written, wherein the data partition to be written is any one of the multiple data partitions, the data bucket to be written is any one of the multiple data buckets included in the data partition to be written, each of the data partitions includes a plurality of column clusters divided by columns, each of the column clusters is used for writing characteristic data of multiple associated types, and the column clusters are physically stored independently;

[0014] The receiving module is further used to receive a range query request, wherein the range query request includes multiple types to be queried;

[0015] The reading module is used to read the characteristic data of each row in each column cluster of the multiple types to be queried from at least one of the data partitions.

[0016] An embodiment of the present application provides an electronic device, including:

[0017] A memory for storing executable instructions;

[0018] The processor is used to implement the data processing method of the data table provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing a data processing method for a data table provided in an embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions, which are used to implement the data processing method of the data table provided in the embodiment of the present application when executed by a processor.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] The data table is divided into multiple data partitions by rows, each data partition is divided into multiple data buckets by rows, and each data partition is divided into multiple column clusters by columns, wherein each column cluster is used to write feature data of multiple associated types, and each column cluster is physically stored independently. In this way, since the features in each column cluster are stored in columnar form, the storage efficiency is improved. At the same time, since the column clusters are physically independent, the column clusters corresponding to specific features can be scanned on demand without reading the entire row of the data table, which saves bandwidth, and multiple associated features can be read from the column cluster at one time, thereby improving the range scanning capability and access performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of the architecture of a data processing system 100 for a data table provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application;

[0025] Figure 3 It is a flowchart of a data processing method of a data table provided in an embodiment of the present application;

[0026] Figure 4 It is a flowchart of a data processing method of a data table provided in an embodiment of the present application;

[0027] Figure 5 It is a flowchart of a data processing method of a data table provided in an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of a sparse table data model provided in an embodiment of the present application;

[0029] Figure 7 It is a schematic diagram of the storage model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0034] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0035] 1) Data table: abbreviated as table, it is the basic unit of organizing data in the database. A data table consists of rows and columns.

[0036] 2) Tablet: Partition is the smallest unit of data scheduling, which can be obtained by dividing the data table by row. Data partitions are physically stored independently, that is, a data partition cannot be deployed to different servers. When the load of the server where a data partition is located is too high, the data partition can be split according to certain rules.

[0037] 3) Data bucket (Partition): abbreviated as bucket, is a logical unit obtained by dividing the data table by rows. Multiple consecutive data buckets can form a data partition.

[0038] 4) Row key (Sort_Key): Each row in the data table can represent a data object, and each row can be identified by a row key, that is, different rows in the data table can be distinguished by the row key.

[0039] 5) Column Family: a collection of columns. Different column families are stored independently. For example, different column families can be stored on different disks, so that different disks can be mixed. In addition, columns in a column family can be sorted lexicographically.

[0040] 6) Write Ahead Log (WAL): is a series of technologies used to provide atomicity and persistence in relational database systems. In a system using WAL, all data must be written to a log file before being submitted.

[0041] 7) Log Structure Merge Tree (LSM): It is a hierarchical, ordered data structure based on hard disk storage. LSM tree includes memory table (MemTable), immutable memory table (ImmutableMemTable) and sorted string table (SSTable). The memory table is a data structure in memory, which is used to store the most recently updated data. The data is organized in order according to the key. Because the data is temporarily stored in the memory, the memory is not a reliable storage. If the power is off, the data will be lost. Therefore, the reliability of the data is usually guaranteed by WAL. When the memory table reaches a certain size, it will be converted into an immutable memory table. The immutable memory table is an intermediate state of converting the memory table into a sorted string table. The write operation is processed by the new memory table, and the data update operation is not blocked during the transfer process. The sorted string table is an ordered set of key-value pairs and is the data structure of the LSM tree on the disk. In order to speed up the reading of the sorted string table, the key search can be accelerated by establishing a key index and a Bloom filter.

[0042] 8) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0043] The embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium, and computer program product for a data table, which can take into account both efficient storage and high-performance range scanning requirements. The electronic device provided by the embodiments of the present application is described below. The electronic device provided by the embodiments of the present application can be implemented as a server, or can be implemented collaboratively by a server and a terminal device.

[0044] See also Figure 1 , Figure 1 1 is a schematic diagram of the architecture of a data processing system 100 for a data table provided in an embodiment of the present application, in order to realize applications that support both efficient storage and high-performance range scanning requirements, such as Figure 1As shown, the data processing system 100 of the data table includes: a server 200 (for example, it can be a distributed data storage system, a data management system, etc.), a network 300 and a terminal device 400, wherein the network 300 can be a local area network or a wide area network, or a combination of the two, the terminal device 400 is a terminal device associated with a user (for example, a provider of business data), the terminal device 400 can also be called a business terminal device, and a client 410 runs on the terminal device 400. The client 410 can be various types of clients, for example, it can be a client provided by a distributed data storage system, a client provided by a data management system, a browser, etc.

[0045] In some embodiments, the server 200 may divide the data table into multiple data partitions by row, and divide each data partition into multiple data buckets by row, and then the server 200 may also divide each data partition into multiple column clusters by column, wherein each column cluster may be used to write feature data of multiple associated types, and the column clusters are physically stored independently. Subsequently, after the server 200 receives the business data uploaded by the user through the terminal device 400, the business data may be written as a row into the data bucket to be written in the data partition to be written. When the server 200 subsequently receives a range query request carrying multiple types to be queried sent by the terminal device 400, the multiple types to be queried may be first extracted from the range query request, and the feature data of each row in each column cluster including the multiple types to be queried may be read from at least one data partition, and finally, the server 200 may return the read feature data to the terminal device 400, so that the terminal device 400 calls the human-computer interaction interface of the client 410 for display.

[0046] In other embodiments, the embodiments of the present application can also be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0047] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0048] For example, Figure 1The server 200 in the example may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal device 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal device 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0049] The structure of the electronic device provided in the embodiment of the present application is further described below. Taking the electronic device as a server as an example, see Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, Figure 2 The electronic device 500 shown includes: at least one processor 510, a memory 540, and at least one network interface 520. The various components in the electronic device 500 are coupled together via a bus system 530. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 530 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 530 is not described in detail. Figure 2 Various buses are labeled as bus system 530 .

[0050] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0051] The memory 540 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 540 may optionally include one or more storage devices that are physically remote from the processor 510.

[0052] The memory 540 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 540 described in the embodiments of the present application is intended to include any suitable type of memory.

[0053] In some embodiments, memory 540 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0054] Operating system 541, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0055] A network communication module 542, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0056] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The data processing device 543 of the data table stored in the memory 540 is shown, which can be software in the form of programs and plug-ins, including the following software modules: receiving module 5431, writing module 5432, reading module 5433, obtaining module 5434, dividing module 5435, deploying module 5436, restoring module 5437 and storing module 5438. These modules are logical, so they can be arbitrarily combined or further divided according to the functions implemented. It should be pointed out that in Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the data processing device 543 in the data table excludes the implementation that can only include the receiving module 5431, the writing module 5432 and the reading module 5433. The functions of each module will be explained below.

[0057] The data processing method of the data table provided in the embodiment of the present application will be specifically described below in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0058] See also Figure 3 , Figure 3 is a flow chart of a data processing method for a data table provided in an embodiment of the present application, which will be combined with Figure 3The steps shown are explained.

[0059] In step 101, business data to be written into a data table is received.

[0060] Here, the business data may be various types of data, such as document data, order data, browsing data, etc., and the embodiment of the present application does not specifically limit the type of business data. In addition, the data table may include multiple data partitions divided by rows, and each data partition may include multiple data buckets divided by rows.

[0061] In some embodiments, before receiving a piece of business data to be written into a data table, the data table can be pre-divided into multiple data partitions evenly (or randomly) by row, and each data partition can be evenly (or randomly) divided into multiple data buckets by row.

[0062] For example, taking the data table as an order table, when there are 5 different types of orders, the order table can be divided into 5 corresponding data partitions by row, for example, including data partition 1 corresponding to fruit orders, assuming that data partition 1 includes rows 1 to 1000 of the order table, data partition 2 corresponding to food orders, assuming that data partition 2 includes rows 1001 to 4000 of the order table, data partition 3 corresponding to clothing orders, assuming that data partition 3 includes rows 4001 to 5000 of the order table, data partition 4 corresponding to home appliance orders, assuming that data partition 4 includes rows 5001 to 6000 of the order table, and data partition 5 corresponding to toy orders, assuming that data partition 5 includes rows 6001 to 10000 of the order table. In addition, for each data partition, it can be further divided into multiple data buckets by row. For example, taking data partition 1 corresponding to fruit orders as an example, data partition 1 can be further divided into data bucket 1 corresponding to local fruits, assuming that data bucket 1 includes rows 1 to 200 of the order table, data bucket 2 corresponding to foreign fruits, assuming that data bucket 2 includes rows 201 to 300 of the order table, and data bucket 3 corresponding to imported fruits, assuming that data bucket 3 includes rows 301 to 1000 of the order table.

[0063] It should be noted that the number of rows included in different data partitions may be the same or different. For example, the number of rows included in all data partitions in the data table may be the same (i.e., evenly divided), or the number of rows included in all data partitions is different (i.e., randomly divided), or some data partitions include the same number of rows, and some data partitions include a different number of rows. In addition, the number of rows included in different data buckets may be the same or different. For example, the number of rows included in multiple data buckets under the same data partition may be the same (i.e., evenly divided), and the number of rows included in data buckets under different data partitions is different. Of course, the number of rows included in different data buckets under the same data partition may also be different (i.e., randomly divided), and the embodiments of the present application do not make specific limitations on this.

[0064] In step 102, the business data is written as a row into the to-be-written data bucket in the to-be-written data partition.

[0065] Here, the data partition to be written can be any one of the multiple data partitions (that is, the data partition to be written is included in the multiple data partitions), the data bucket to be written can be any one of the multiple data buckets included in the data partition to be written (that is, the data bucket to be written is included in the multiple data buckets of the data partition to be written), each data partition may include multiple column clusters divided by columns, each column cluster can be used to write characteristic data of multiple associated types, and the column clusters are physically stored independently, that is, the same column cluster can only be deployed in one server and cannot be deployed in different servers.

[0066] It should be noted that writing the business data as a row into the data bucket to be written in the data partition to be written means: writing multiple different types of feature data included in the business data into the same row of the data bucket to be written. For example, taking the business data as document 1, multiple different types of feature data of document 1 (such as reading volume, like volume, forwarding volume, quality score, etc.) can be written into the same row (such as the first row in data bucket 1) of the data bucket to be written (such as data bucket 1).

[0067] In addition, it should be noted that the feature data of the above-mentioned multiple associated types may refer to feature data whose correlation is greater than the correlation threshold. That is to say, when the correlation between multiple feature data is greater than the correlation threshold, continuous reading and writing are required, and the feature data that need to be continuously read and written can be written into the same column cluster.

[0068] In some embodiments, taking multiple data partitions as 5 data partitions as an example, assume that they are data partition 1, data partition 2, data partition 3, data partition 4 and data partition 5 respectively, wherein data partition 1 includes 3 data buckets, assume that they are data bucket 1, data bucket 2 and data bucket 3 respectively, data partition 2 includes 2 data buckets, assume that they are data bucket 4 and data bucket 5 respectively, data partition 3 includes 4 data buckets, assume that they are data bucket 6, data bucket 7, data bucket 8 and data bucket 9 respectively, data partition 4 includes 2 data buckets, assume that they are data bucket 10 and data bucket 11 respectively, data partition 5 includes 3 data buckets, assume that they are data bucket 12, data bucket 13 and data bucket 14 respectively. At the same time, assuming that the data bucket 12 included in the current data partition 5 is in an idle state (i.e., no data has been written), and the data buckets included in other data partitions have all been written with data, the received business data can be written as a row into the data bucket 12 (i.e., the data bucket to be written) in the data partition 5 (i.e., the data partition to be written). For example, multiple different types of feature data included in the business data can be written into the third row of the data bucket 12.

[0069] In other embodiments, see Figure 4 , Figure 4 is a flow chart of a data processing method for a data table provided in an embodiment of the present application, such as Figure 4 As shown, Figure 3 Step 102 shown may be performed by Figure 4 Steps 1021 to 1023 shown in the figure are implemented by combining Figure 4 The steps shown are explained.

[0070] In step 1021, the hash values ​​of the addresses of different servers are mapped to different positions in the hash ring.

[0071] In some embodiments, multiple data partitions can be deployed in different servers, and the addresses of different servers can be obtained, and the addresses of each server can be hashed to obtain the hash values ​​corresponding to the addresses of different servers. Subsequently, the hash values ​​of the addresses of different servers can be mapped to different positions in the hash ring. Among them, the hash value is a unique and extremely compact numerical representation of a piece of data. For example, taking server 1 as an example, after obtaining the address of server 1, a hash algorithm (such as MD2, MD5, SHA-1, etc.) can be used to calculate the hash value corresponding to the address of server 1.

[0072] For example, taking the multiple data partitions as 5 data partitions, assuming that they are data partitions 1 to 5, the 5 data partitions can be deployed in different servers, for example, data partition 1 can be deployed in server 1, data partition 2 can be deployed in server 2, data partition 3 can be deployed in server 3, data partition 4 can be deployed in server 4, and data partition 5 can be deployed in server 5. Then, the addresses of the 5 servers, such as Internet Protocol (IP) addresses, can be obtained respectively, and the obtained IP addresses can be hashed, for example, the MD5 algorithm can be used to perform hash operations on the IP addresses of the 5 servers to obtain hash values ​​corresponding to the IP addresses of the 5 servers. Finally, the hash values ​​of the IP addresses of these five servers can be mapped to different positions in a hash ring that is connected end to end (assuming that the hash ring includes 8 positions, namely position 1 to position 8). For example, the hash value of the IP address of server 1 (assuming hash value 1) can be mapped to position 1 in the hash ring, the hash value of the IP address of server 2 (assuming hash value 2) can be mapped to position 2 in the hash ring, the hash value of the IP address of server 3 (assuming hash value 3) can be mapped to position 3 in the hash ring, the hash value of the IP address of server 4 (assuming hash value 4) can be mapped to position 4 in the hash ring, and the hash value of the IP address of server 5 (assuming hash value 5) can be mapped to position 5 in the hash ring.

[0073] In step 1022, the address of the business end device that submits the business data is mapped into the hash ring. Starting from the business end device mapping position of the address of the business end device in the hash ring, the search is conducted in a clockwise or counterclockwise direction, and the server corresponding to the first server mapping position found is used as the server to be written.

[0074] In some embodiments, following the above, after the hash values ​​of the IP addresses of the five servers are mapped to different positions in the hash ring, the IP address of the business-end device (such as a computer) that submits the business data can be further obtained, and after the IP address of the business-end device is obtained, the IP address of the business-end device is also mapped to the hash ring, and starting from the business-end device mapping position of the IP address of the business-end device in the hash ring, the search is performed in a clockwise or counterclockwise direction, and the server corresponding to the first server mapping position found is used as the server to be written. For example, taking the business-end device mapping position of the IP address of the business-end device in the hash ring as position 7 in the hash ring, the search is performed in a clockwise direction. Assuming that position 5 in the hash ring is the closest position to position 7 in the clockwise direction, the server corresponding to position 5 (i.e., server 5) can be used as the server to be written.

[0075] It should be noted that after obtaining the IP address of the business-end device, you can also first perform a hash operation on the IP address of the business-end device to obtain the hash value of the IP address of the business-end device, and then map the hash value of the IP address of the business-end device to the corresponding position in the hash ring. The embodiment of the present application does not make specific limitations on this.

[0076] In step 1023, at least one data partition deployed on the to-be-written server is determined as the to-be-written data partition, and the business data is taken as a row and written into the to-be-written data bucket in the to-be-written data partition.

[0077] In some embodiments, continuing from the above, after obtaining the server to be written (for example, server 5) based on the hash ring, at least one data partition deployed on server 5 can be determined as the data partition to be written. Since only data partition 5 is deployed on server 5, data partition 5 can be directly used as the data partition to be written. Then, the business data received in step 101 can be taken as a row and written into the data bucket to be written in data partition 5 (i.e., the data partition to be written). For example, assuming that data partition 5 includes 3 data buckets, namely data bucket 12, data bucket 13 and data bucket 14, among which data bucket 13 has no data written, the received business data can be written into data bucket 13 included in data partition 5. In this way, by deploying different data partitions in different servers, load balancing between servers can be achieved.

[0078] It should be noted that when multiple data partitions are deployed on the server to be written, any data partition can be used as the data partition to be written, or the data write rate of each data partition can be determined first, that is, the amount of data currently written to each data partition is determined to account for the total amount of data that can be written to the data partition, and the data partition with the smallest data write rate is used as the data partition to be written. For example, taking the server to be written as server 1, there are 3 data partitions deployed on server 1, assuming that they are data partition 1, data partition 2 and data partition 3 respectively, among which the current data write rate of data partition 1 is 80%, the current data write rate of data partition 2 is 90%, and the current data write rate of data partition 3 is 85%. In this case, data partition 1 can be used as the data partition to be written, thereby avoiding writing a large amount of business data into the same data partition. Of course, a data partition that matches the type of business data in multiple data partitions can also be used as a data partition to be written. For example, assuming that data partition 1 is used to store data for fruit orders, data partition 2 is used to store data for beverage orders, and data partition 3 is used to store data for clothing orders, and assuming that the received business data is the order data for clothing A, data partition 3 can be used as the data partition to be written. The embodiment of the present application does not make specific limitations on this.

[0079] In some embodiments, multiple adjacent column clusters in a data bucket can share a write-ahead log; when business data intersects with multiple column clusters to be written in the data bucket to be written, that is, when the business data needs to be written to multiple column clusters, the following processing can also be performed: multiple instructions for writing business data in multiple column clusters to be written are stored in the write-ahead log, the business data is written into the data cache of the multiple column clusters to be written, and the data cache is synchronized to the server where the data bucket is located to enable the server to store persistently. For example, the server can store the business data in a hard disk to achieve persistent storage; in response to a failure of the server where the data bucket is located, data recovery is performed on the multiple column clusters to be written based on the write-ahead log.

[0080] For example, taking multiple adjacent column clusters as column clusters 1 to 3, column clusters 1 to 3 can share a write-ahead log. When business data needs to be written to column clusters 1 to 3, multiple instructions for writing business data in column clusters 1 to 3 can be first stored in a log file, and the business data can be written into the data cache of column clusters 1 to 3, and the data cache can be synchronized to the server where the data bucket is located, so that the server stores the business data in the hard disk. Subsequently, when it is detected that the server where the data bucket is located has a failure (such as a sudden power outage), the log file can be checked, and the multiple instructions previously stored can be obtained from the log file, and data recovery can be performed on column clusters 1 to 3 based on the multiple instructions obtained.

[0081] That is to say, in a system that uses a write-ahead log, all data modifications must be written to a log file before being submitted. The log file usually includes a redo log and an undo log. Whenever an operation is performed, the operation can be written to the redo log before the data is changed. In this way, when a failure such as a power outage occurs, the operation can be continued after restart. In addition, when some changes cannot be completed halfway through execution, the undo log can be used to restore to the state before the change. That is, the redo log can repair data when a server fails to ensure the persistence of the transaction, while the undo log can ensure the atomicity of the transaction.

[0082] In other embodiments, each type of feature of business data may occupy multiple columns in the column cluster to be written into the data bucket, wherein one column is used to store the feature value of the feature and at least two columns are used to store metadata of the feature (i.e., information used to describe the feature).

[0083] For example, taking business data as documents, the characteristic values ​​(such as the document's category, quality, etc.) of document features (such as the document's category, quality, etc.) can be written into column 1 of the column cluster of the data bucket to be written, and the meta-information of the document features can be written into columns 2 and 3 of the column cluster. The meta-information of the document features refers to the information used to describe the document features. For example, when the document feature is document quality, the corresponding meta-information can be information such as the quality evaluation dimension and the number of people involved in the evaluation.

[0084] In some embodiments, the column cluster to be written into the data bucket may also include a secondary index, wherein the secondary index includes multiple associated data of features in the business data (such as multiple articles published by the same account); the secondary index may occupy one of the multiple columns alone, or share a column with the feature value.

[0085] For example, taking the data bucket to be written as data bucket 1, the column cluster of data bucket 1 can also include a secondary index, wherein the secondary index can include multiple related data of the features in the business data (such as all articles published by a certain account), and the secondary index can occupy a row of the multiple columns included in the column cluster alone to achieve decoupling from the feature value; of course, it can also share a column with the feature value, that is, store the secondary index and the feature value in the same column to improve storage efficiency.

[0086] In other embodiments, see Figure 5 , Figure 5 is a flow chart of a data processing method for a data table provided in an embodiment of the present application, such as Figure 5 As shown, after executing Figure 3 After step 102 shown, you can also perform Figure 5 Step 105 shown combines Figure 5 The steps shown are explained.

[0087] In step 105, the business data is stored in a storage view of the data table in the form of key-value pairs.

[0088] Here, the key in each key-value pair includes the following sorting keys for each type of characteristic data in the business data: the data bucket identifier of the data bucket (used to identify the uniquely corresponding data bucket), the column key of the column, and the row key of the row; the value in the key-value pair is a type of characteristic data in the business data; and, empty data units in the data table are not assigned key-value pairs.

[0089] In some embodiments, in order to further improve the storage and access bandwidth utilization, after writing the business data as a row into the data bucket to be written in the data partition to be written, the business data can also be compressed. In the storage view of the data table, the business data can be stored in the form of key-value pairs after compression, wherein the key in each key-value pair includes the following sorted values ​​of each type of characteristic data in the business data: the data bucket identifier of the data bucket, the column key of the column, and the row key of the row; the value in the key-value pair is a type of characteristic data in the business data. In addition, when the data unit is empty, it will not be stored, that is, the empty data unit in the data table is not assigned a key-value pair. In this way, when storing and accessing large-scale sparse data, since the empty data unit is not stored, it can greatly save storage space and improve bandwidth utilization.

[0090] In other embodiments, each data unit in the data table can be identified by quaternary information, wherein the quaternary information includes: a data bucket identifier (e.g., a data bucket key) of the data bucket where the data unit is located, a column cluster identifier of the column cluster where the data unit is located, a row key of the row where the data unit is located, and a column key of the column where the data unit is located.

[0091] For example, taking data unit 1 as an example, the characteristic data stored in data unit 1 can be uniquely determined by four pieces of information (e.g., a quadruple), including the data bucket identifier of the data bucket where data unit 1 is located (used to determine which data bucket in the data table data unit 1 is located in), the column cluster identifier of the column cluster where data unit 1 is located (used to determine which column cluster in the data table data unit 1 is located in), the row key of the row where data unit 1 is located (used to determine which row in the data bucket data unit 1 is located in), and the column key of the column where data unit 1 is located (used to determine which column in the column cluster data unit 1 is located in).

[0092] In step 103, a range query request is received.

[0093] Here, the range query request may include multiple types to be queried. That is, the range query request is a query request carrying multiple types to be queried.

[0094] In some embodiments, when a selection operation for multiple types is received from a user, the multiple types selected by the user may be used as multiple types to be queried, and the multiple types to be queried may be added to the range query request.

[0095] In step 104, feature data of each row in each column cluster including multiple types to be queried is read from at least one data partition.

[0096] In some embodiments, after receiving a range query request, multiple types to be queried carried by the range query request can be first extracted from the range query request, and then the characteristic data of each row in each column cluster including the multiple types to be queried can be read from at least one data partition. That is, by dividing the data table into multiple data partitions by rows, and dividing each data partition into multiple data buckets by rows, and dividing each data bucket into multiple column clusters by columns, since each column cluster is stored independently and the characteristic data in each column cluster is stored in columnar form, at least one data partition can be scheduled to realize range scanning by feature search, thereby improving the range scanning capability and access performance.

[0097] It should be noted that the number of at least one data partition here depends on the query range of the range query request, that is, the range of the row key to be queried. In other words, the larger the range of the row key to be queried, the larger the number of at least one data partition, and the smaller the range of the row key to be queried, the smaller the number of at least one data partition.

[0098] In other embodiments, the range query request may further include multiple row keys to be searched. In this case, after reading the feature data in at least one column cluster including multiple types to be queried, the following processing may be performed: retrieving the feature data in the rows corresponding to the multiple row keys to be searched from the read feature data.

[0099] For example, when the range query request also carries multiple row keys to be searched, after reading the feature data in at least one column cluster including multiple types to be queried, the feature data in the rows corresponding to the multiple row keys to be searched can also be retrieved from the read feature data. For example, taking the multiple row keys to be searched as row key 1 and row key 2 as an example, where row key 1 is the row key corresponding to the first row in the data bucket, and row key 2 is the row key corresponding to the second row in the data bucket, the feature data of the rows corresponding to row key 1 and row key 2 can be retrieved from the read feature data, for example, the feature data in the first row and the second row in the data bucket can be read to meet more fine-grained retrieval requirements.

[0100] It should be noted that when the range query request does not include multiple row keys to be queried, after reading the feature data in at least one column cluster including multiple types to be queried, the feature data in all rows in the column cluster can be directly retrieved from the read feature data. In other words, when the user needs to retrieve the feature data of certain rows in a more refined manner, the corresponding row key can be added to the range query request; when the user does not add the row key to be queried in the range query request, the feature data in all rows will be retrieved, that is, the user can perform flexible searches according to their own needs, which improves the user experience.

[0101] In some embodiments, the following processing can also be performed: receiving a single query request, wherein the single query request includes an identifier of the business data to be queried; obtaining a row key to be queried corresponding to the identifier of the business data to be queried; and reading characteristic data located in the row corresponding to the row key to be queried from the data table.

[0102] For example, the data table can also be associated with an index table, and the index table can include a mapping relationship between the following elements: a business data identifier of the business data, and a row key of the row where the business data is located, wherein the row key of the row where the business data is located is a characteristic parameter of the business data. The above-mentioned acquisition of the row key to be queried corresponding to the business data identifier to be queried can be achieved in the following manner: using the business data identifier to be queried as the corresponding row key to be queried, or querying the row key to be queried corresponding to the business data identifier to be queried from the index table. In other words, the business data identifier to be queried (such as a document identifier) ​​can be directly stored as the row key to be queried, so that the business data identifier to be queried can be directly used as the corresponding row key to be queried later. Of course, a mapping relationship between the business data identifier to be queried and the row key to be queried can also be established, and the mapping relationship between the two can be stored in the index table, so that the row key to be queried corresponding to the business data identifier to be queried can be queried from the index table later, and the embodiments of the present application do not make specific limitations on this.

[0103] In other embodiments, multiple data partitions can be deployed in different servers, and the following processing can also be performed: in response to the load parameter of the server where any data partition is located being higher than the load parameter threshold, any data partition is evenly (or randomly) divided into multiple new data partitions by row; and the multiple new data partitions are deployed to different servers.

[0104] For example, take the case where the multiple data partitions are 5 data partitions, assuming that they are data partition 1, data partition 2, data partition 3, data partition 4 and data partition 5 respectively. These 5 data partitions can be deployed in different servers. For example, data partition 1 can be deployed in server 1, data partition 2 can be deployed in server 2, data partition 3 can be deployed in server 3, data partition 4 can be deployed in server 4, and data partition 5 can be deployed in server 5. Taking data partition 2 as an example, when it is detected that the load parameter (such as CPU occupancy rate, memory occupancy rate) of the server where data partition 2 is located (i.e., server 2) is higher than the load parameter threshold, for example, when the memory occupancy rate of server 2 is greater than the occupancy threshold, data partition 2 can be evenly divided into multiple new data partitions by row (the number of rows included in the new data partition is less than the number of rows included in data partition 2). For example, data partition 2 (assuming that data partition 2 includes rows 1000 to 4000 of the data table) can be evenly divided into 3 new data partitions by row, assuming that they are new data partition 1 (assuming that it includes rows 1000 to 4000 of the data table). The new data partition 1 (assuming it includes rows 2001 to 3000 of the data table), the new data partition 2 (assuming it includes rows 2001 to 3000 of the data table), and the new data partition 3 (assuming it includes rows 3001 to 4000 of the data table), and the three new data partitions are deployed in different servers respectively. For example, the new data partition 1 can be deployed in server 1, the new data partition 2 can be deployed in server 3, and the new data partition 3 can be deployed in server 4. In this way, when the load of the server where a data partition is located is too high, the data partition can be split into multiple new data partitions, and scheduling can be performed on multiple servers to improve access performance.

[0105] The data processing method for a data table provided in an embodiment of the present application divides the data table into multiple data partitions by rows, and divides each data partition into multiple data buckets by rows, and divides each data bucket into multiple column clusters by columns, wherein each column cluster is used to write feature data of multiple associated types, and each column cluster is physically stored independently. In this way, since the features in each column cluster are stored in columnar form, the storage efficiency is improved; at the same time, since the column clusters are physically independent, the column clusters corresponding to specific features can be scanned on demand without reading the entire row of the data table, which saves bandwidth, and multiple associated features can be read from the column cluster at one time, thereby improving the range scanning capability and access performance.

[0106] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0107] The embodiment of the present application provides a data processing method for a data table, which can achieve high-performance, high-scalability, and high-availability data storage and access capabilities. Specifically, by adopting a sparse table data model, column storage + data bucketing (i.e., dividing the data table into multiple data partitions horizontally, and further dividing each data partition into multiple data buckets, and dividing it into multiple column clusters vertically, storing multiple related features in the same column cluster, each column cluster is stored independently, and the features in each column cluster are stored in columns, so that at least one data partition can be scheduled to implement a range scan by feature search, thereby improving the range scanning capability and access performance), automatic splitting (i.e., splitting a data partition into multiple data partitions, thereby reducing the partition request load and improving access performance), compression (i.e., compressing multiple data before writing data to reduce data storage and storage costs), etc., the storage and access bandwidth utilization is improved. On this basis, the embodiment of the present application also provides a new table architecture (i.e., a dual table structure of a data table and an index table,

[0108] Among them, the index table can be used to store the mapping relationship between the business data identifier and the row key. The mapping relationship is described by the index table so that the row key can include more features of the business data, ensuring that multiple related feature data are physically stored continuously to provide higher range scanning capabilities). Combined with the WAL update mechanism (i.e., multiple adjacent column clusters share one WAL, and success is fed back only when all feature data in different column clusters are written successfully, thereby improving the accuracy of data writing), custom load balancing (including consistent hashing, segmented consistent hashing and other load balancing strategies), and data segmentation mechanism (i.e., splitting a data partition into multiple data partitions) to ensure data correctness, it also provides high-performance and highly scalable range scanning capabilities (range scanning is data reading based on multiple features. Since column identifiers correspond to features, column storage can enable data under the same feature to be read continuously, thereby improving the efficiency of range scanning). It is an efficient solution for massive data storage and access.

[0109] The data processing method of the data table provided in the embodiment of the present application is described in detail below.

[0110] In some embodiments, see Figure 6 , Figure 6 is a schematic diagram of a sparse table data model provided in an embodiment of the present application, such as Figure 6As shown, the data is logically stored in a table form, and is divided into several data buckets (Partition) in the horizontal direction (i.e., the direction corresponding to the rows in the data table) according to the Hash-Range partitioning strategy. Each data bucket includes multiple rows, and each row of data is uniquely located by two keys: the data bucket identifier (Partition_Key) and the row key (Sort_Key). In addition, the data in the data bucket can be sorted in order according to the row key, and multiple data buckets are unordered and have no intersection. Multiple consecutive data buckets can form a data partition (Tablet), for example, data bucket 100 and data bucket 101 can be combined into a data partition (i.e., data partition_100-101). The data partition is the smallest unit of data scheduling and is physically stored independently. When the partition request load is greater than the load threshold (i.e., the load is too high), for example, when the memory occupancy rate of the server where a data partition is located is greater than the occupancy rate threshold, the data partition can be split according to certain rules, for example, a large data partition can be split into two small data partitions and deployed in two servers respectively to improve access performance.

[0111] Continue to see Figure 6 In the vertical direction (i.e. the direction corresponding to the columns of the data table), multiple columns (Qualifier) ​​can form a column family (Column Family), and a data partition can include multiple column families. In addition, all column families in the same data partition can share a WAL, thereby ensuring the atomicity and integrity of data written in different column families, that is, when writing data in different column families of a data partition, only when all column families are written successfully can the data be reported successfully, thereby ensuring the integrity of the written data. In addition, in the event of a crash, data can be restored with the help of WAL to avoid data loss. In addition, different column families can use the LSM structure to store data independently, and data that is often read and written together (such as feature data that needs to be read and written continuously, such as when the correlation between multiple feature data is greater than the correlation threshold, continuous reading and writing is required) can be put into the same column family for storage. Since different column families are physically independent of each other, the column family corresponding to a specific feature can be scanned on demand without reading the entire row of the data table, saving bandwidth, and multiple related feature data can be read from the column family at one time, thereby providing higher access performance.

[0112] In addition, each data cell (Cell) in the data table can be uniquely identified by the four-tuple of <data partition identifier (Partition_Key), row key (Sort_Key), column family (Column Family), column identifier (Qualifier_ID)>. In addition, due to the use of column families, the row keys of rows in different column families can be repeated. For example, the row keys in each column family can be counted starting from 0. In addition, in the physical view of storage, the data can be stored in the form of key-value pairs (KV) after compression, and the data cell will not be stored when it is empty (Null). In addition, when storing and accessing large-scale sparse data, due to the use of LSM data structure, empty data cells are not stored, which can greatly save storage space and improve bandwidth utilization.

[0113] In addition, in terms of storage sorting, business data can be stored in the sorted order of <data bucket identifier (Partition_Key), column identifier (Qualifier_ID), and row key (Sort_Key)>, so that multiple related feature data can be stored in adjacent columns. When a range scan is performed on some features, since multiple related feature data are physically stored continuously (that is, stored in the same column cluster), access bandwidth can be used more effectively and higher scanning performance can be provided.

[0114] In other embodiments, Figure 6 Based on the sparse table data model shown, the present application embodiment also provides Figure 7 The storage model shown in FIG. 1 shows a storage model in which data storage includes two tables, namely, a data table and an index table (the index table is used for single query, i.e., querying feature data in a row, and cannot be used for range scanning). The data table includes multiple column clusters, each of which stores multiple features. Each feature occupies multiple columns in the same column cluster, which are used to store the following fields of the feature: original feature value (occupying one column), feature meta information (occupying multiple columns), and secondary index (Sec_Index), wherein the secondary index is optional and can occupy a column alone or share a column with the original feature value. The index table is used to store the mapping information between the business data identifier and the row key (e.g., the mapping relationship between the document identifier and the row key).

[0115] Continue to see Figure 7, a business data (such as a document) can include multiple different types of features. Each business data has a unique corresponding business data identifier (Doc_ID) on the business side. The business data can be stored as a row in the data table (that is, multiple features included in the business data are stored in the same row of the data table), and can be uniquely identified by the row key. In most cases, the business data identifier (such as the document identifier) ​​can be stored as the row key. In the extended scenario, the row key can include certain characteristics of the document (such as publication time, account number, etc.), ensuring that data with the same characteristics are physically stored continuously to provide higher range scanning capabilities. Therefore, an index table is required to save the unique mapping from document identifier to row key to ensure that the data can be accessed correctly.

[0116] It should be noted that index information can be stored in both index tables and data tables, but their functions are different. The index table stores a unique mapping, providing the ability to access data through document identifiers. The secondary index in the data table is a non-unique mapping, which can provide the ability to quickly access a batch of data based on a certain feature value, such as accessing all articles published under a certain account, to speed up access performance in specific scenarios. In principle, the secondary index can share a column with the original feature value, or it can occupy a column independently. When it occupies a column independently, it is decoupled from the original feature value, and the storage of the original feature value can be more flexible, and more complex data structures can be stored, such as feature value stress testing, serialization, and nesting of more information.

[0117] In addition, when business data (such as documents) are accessed, they can be written to different data partitions of the data table through load balancing strategies such as consistent hashing and segmented consistent hashing. Figure 7 Each logic shown in is a data bucket, where logic_0 to logic_10 constitute data partition_1_10). Multiple columns of feature data (such as original feature values, feature metadata, etc.) can be written into the data table, and the integrity and correctness of the feature data can be guaranteed through the WAL atomic update mechanism. When accessing features, the underlying pure column-based sorting storage is used, and access is performed on demand when scanning a large range. That is, according to the range of business data increased by the business end (if not specified, the feature data in all rows can be scanned by default) and the feature range (that is, multiple features that need to be scanned), the corresponding column cluster identifier can be found through the mapping relationship between the feature and the column cluster identifier, and the feature data in all rows in the column cluster corresponding to the column cluster identifier can be read, and the feature data within the row key range can be retrieved from it. Since the entire row does not need to be read, the bandwidth utilization is greatly improved compared to row storage. When scanning a large range, at least one data partition is scanned at the same time (the number of data partitions depends on the scanning range, that is, the range of the row key), combined with automatic partition splitting, the load of a single server is reduced, thereby achieving high-performance range scanning requirements.

[0118] In summary, the data processing method for the data table provided in the embodiment of the present application combines data compression, columnar sparse storage, data segmentation, automatic splitting and other technologies to reduce storage costs and improve bandwidth utilization. In addition, by horizontally expanding service capabilities through data bucketing, high-performance scanning capabilities are provided while reducing the possibility of downtime. Data is stored in the same table, combined with the WAL update mechanism, to ensure the correctness of the data. Based on the above optimization effects, in actual business scenarios, the storage and access costs of data are greatly reduced.

[0119] The following further describes an exemplary structure of the data table data processing device 543 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example Figure 2 As shown, the software modules in the data processing device 543 of the data table stored in the memory 540 may include: a receiving module 5431 , a writing module 5432 and a reading module 5433 .

[0120] The receiving module 5431 is used to receive business data to be written into a data table, wherein the data table includes multiple data partitions divided by rows, and each data partition includes multiple data buckets divided by rows; the writing module 5432 is used to write the business data as a row into the data bucket to be written in the data partition to be written, wherein the data partition to be written is any one of the multiple data partitions, the data bucket to be written is any one of the multiple data buckets included in the data partition to be written, each data partition includes multiple column clusters divided by columns, each column cluster is used to write characteristic data of multiple associated types, and the column clusters are physically stored independently; the receiving module 5431 is also used to receive a range query request, wherein the range query request includes multiple types to be queried; the reading module 5433 is used to read the characteristic data of each row in each column cluster including multiple types to be queried from at least one data partition.

[0121] In some embodiments, the range query request also includes multiple row keys to be searched; the reading module 5433 is further used to retrieve the feature data in the rows corresponding to the multiple row keys to be searched from the read feature data after reading the feature data in at least one column cluster including the multiple types to be queried.

[0122] In some embodiments, multiple data partitions are deployed in different servers; the write module 5432 is also used to map the hash values ​​of the addresses of different servers to different positions in the hash ring; map the address of the business-end device that submits the business data to the hash ring, starting from the business-end device mapping position of the address of the business-end device in the hash ring, search in a clockwise or counterclockwise direction, and use the server corresponding to the first server mapping position found as the server to be written; determine at least one data partition deployed on the server to be written as the data partition to be written, and write the business data as a row into the data bucket to be written in the data partition to be written.

[0123] In some embodiments, the receiving module 5431 is also used to receive a single query request, wherein the single query request includes an identifier of the business data to be queried; the data processing device 543 of the data table also includes an acquisition module 5434, which is used to obtain a row key to be queried corresponding to the identifier of the business data to be queried; the reading module 5433 is also used to read characteristic data located in the row corresponding to the row key to be queried from the data table.

[0124] In some embodiments, the data table is associated with an index table, which includes a mapping relationship between the following elements: a business data identifier of the business data, and a row key of the row where the business data is located; wherein the row key of the row where the business data is located is a characteristic parameter of the business data; the acquisition module 5434 is also used to use the business data identifier to be queried as the corresponding row key to be queried, or to query the row key to be queried corresponding to the business data identifier to be queried from the index table.

[0125] In some embodiments, multiple data partitions are deployed in different servers; the data processing device 543 of the data table also includes a partitioning module 5435 and a deployment module 5436, wherein the partitioning module 5435 is used to divide any data partition into multiple new data partitions by rows in response to the load parameter of the server where any data partition is located being higher than the load parameter threshold; the deployment module 5436 is used to deploy multiple new data partitions to different servers.

[0126] In some embodiments, multiple adjacent column clusters in a data bucket share a write-ahead log; when business data intersects with multiple column clusters to be written in the data bucket to be written, the write module 5432 is also used to store multiple instructions for writing business data in the multiple column clusters to be written in the write-ahead log, write the business data into the data cache of the multiple column clusters to be written, and synchronize the data cache to the server where the data bucket is located to enable persistent storage on the server; the data processing device 543 of the data table also includes a recovery module 5437, which is used to recover data for the multiple column clusters to be written based on the write-ahead log in response to a failure of the server where the data bucket is located.

[0127] In some embodiments, the data processing device 543 of the data table also includes a storage module 5438, which is used for storing the business data in the form of key-value pairs in the storage view of the data table after the writing module 5432 writes the business data as a row into the data bucket to be written in the data partition to be written, wherein the key in each key-value pair includes the following sorting keys for each type of characteristic data in the business data: the data bucket identifier of the data bucket, the column key of the column, and the row key of the row; the value in the key-value pair is a type of characteristic data in the business data; and the empty data unit in the data table is not assigned a key-value pair.

[0128] In some embodiments, each type of feature of business data occupies multiple columns in the column cluster to be written into the data bucket, wherein one column is used to store the feature value of the feature and at least two columns are used to store metadata of the feature.

[0129] In some embodiments, the column cluster to be written into the data bucket also includes a secondary index, which includes multiple associated data of features in the business data; the secondary index occupies one of the multiple columns alone, or shares a column with the feature value.

[0130] In some embodiments, each data unit in the data table is determined by quaternary information, which includes: the data bucket identifier of the data bucket where the data unit is located, the column cluster identifier of the column cluster where the data unit is located, the row key of the row where the data unit is located, and the column key of the column where the data unit is located.

[0131] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 3 , Figure 4 ,or Figure 5 The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0132] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer readable storage medium. The processor of the computer device reads the computer executable instruction from the computer readable storage medium, and the processor executes the computer executable instruction, so that the computer device executes the data processing method of the data table described in the embodiment of the present application.

[0133] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the data processing method of the data table provided in the present application embodiment, for example, Figure 3 , Figure 4 ,or Figure 5 The data processing method of the data table is shown.

[0134] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0135] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0136] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0137] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method for a data table, characterized in that: The method comprises: Receive business data to be written into a data table, wherein the data table includes a plurality of data partitions divided by rows, and each of the data partitions includes a plurality of data buckets divided by rows; Writing the business data as a row into a data bucket to be written in the data partition to be written, wherein the data partition to be written is any one of the multiple data partitions, the data bucket to be written is any one of the multiple data buckets included in the data partition to be written, each of the data partitions includes a plurality of column clusters divided by columns, each of the column clusters is used to write characteristic data of multiple associated types, and the column clusters are physically stored independently; Receiving a range query request, wherein the range query request includes multiple types to be queried; From at least one of the data partitions, characteristic data of each row in each column cluster including the multiple types to be queried is read.

2. The method according to claim 1, characterized in that: The range query request also includes multiple row keys to be searched; After reading the feature data in at least one column cluster including the multiple types to be queried, the method further includes: The characteristic data in the rows corresponding to the multiple to-be-searched row keys are retrieved from the read characteristic data.

3. The method according to claim 1 or 2, characterized in that: The multiple data partitions are deployed in different servers; The step of writing the business data as a row into the data bucket to be written in the data partition to be written comprises: Mapping the hash values ​​of the addresses of the different servers to different positions in the hash ring; Mapping the address of the service end device that submits the service data into the hash ring, starting from the service end device mapping position of the address of the service end device in the hash ring, searching in a clockwise or counterclockwise direction, and taking the server corresponding to the first server mapping position found as the server to be written; At least one of the data partitions deployed on the to-be-written server is determined as a to-be-written data partition, and the business data is taken as a row and written into a to-be-written data bucket in the to-be-written data partition.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Receiving a single query request, wherein the single query request includes an identifier of the business data to be queried; Obtain the row key to be queried corresponding to the business data identifier to be queried; The characteristic data located in the row corresponding to the row key to be queried is read from the data table.

5. The method according to claim 4, characterized in that The data table is associated with an index table, and the index table includes a mapping relationship between the following elements: a business data identifier of the business data, and a row key of a row where the business data is located; wherein the row key of the row where the business data is located is a characteristic parameter of the business data; The obtaining the row key to be queried corresponding to the business data identifier to be queried includes: The business data identifier to be queried is used as the corresponding row key to be queried, or the row key to be queried corresponding to the business data identifier to be queried is queried from the index table.

6. The method according to any one of claims 1 to 5, characterized in that: The multiple data partitions are deployed in different servers; The method further comprises: In response to a load parameter of a server where any of the data partitions is located being higher than a load parameter threshold, dividing any of the data partitions into a plurality of new data partitions by row; The multiple new data partitions are deployed to the different servers.

7. The method according to any one of claims 1 to 6, characterized in that: Multiple adjacent column clusters in the data bucket share a write-ahead log; When the business data intersects with a plurality of to-be-written column clusters in the to-be-written data bucket, the method further includes: storing a plurality of instructions for writing the business data in the plurality of to-be-written column clusters into the write-ahead log, writing the business data into a data cache of the plurality of to-be-written column clusters, and synchronizing the data cache to a server where the data buckets are located, so that the server stores the data persistently; In response to a failure of a server where the data bucket is located, data recovery is performed on the multiple to-be-written column clusters based on the write-ahead log.

8. The method according to any one of claims 1 to 7, characterized in that: After writing the business data as a row into the data bucket to be written in the data partition to be written, the method further includes: In the storage view of the data table, the business data is stored in the form of key-value pairs, wherein the key in each key-value pair includes the following sorting keys for each type of characteristic data in the business data: the data bucket identifier of the data bucket, the column key of the column, and the row key of the row; the value in the key-value pair is a type of characteristic data in the business data; and the empty data unit in the data table is not assigned the key-value pair.

9. The method according to any one of claims 1 to 8, characterized in that: Each type of feature of the business data occupies multiple columns in the column cluster to be written into the data bucket, wherein one column is used to store the feature value of the feature, and at least two columns are used to store the meta information of the feature.

10. The method according to claim 9, characterized in that The column cluster to be written into the data bucket also includes a secondary index, and the secondary index includes multiple associated data of the features in the business data; the secondary index occupies one column among the multiple columns alone, or shares a column with the feature value.

11. The method according to any one of claims 1 to 10, characterized in that: Each data unit in the data table is determined by four-element information, and the four-element information includes: The data bucket identifier of the data bucket where the data unit is located, the column cluster identifier of the column cluster where the data unit is located, the row key of the row where the data unit is located, and the column key of the column where the data unit is located.

12. A data processing device for a data table, characterized in that: The device comprises: A receiving module, configured to receive business data to be written into a data table, wherein the data table includes a plurality of data partitions divided by rows, and each of the data partitions includes a plurality of data buckets divided by rows; A writing module, used for writing the business data as a row into a data bucket to be written in the data partition to be written, wherein the data partition to be written is any one of the multiple data partitions, the data bucket to be written is any one of the multiple data buckets included in the data partition to be written, each of the data partitions includes a plurality of column clusters divided by columns, each of the column clusters is used for writing characteristic data of multiple associated types, and the column clusters are physically stored independently; The receiving module is further used to receive a range query request, wherein the range query request includes multiple types to be queried; The reading module is used to read the characteristic data of each row in each column cluster of the multiple types to be queried from at least one of the data partitions.

13. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the data processing method of the data table according to any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the data processing method of the data table according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instruction is executed by a processor, the data processing method of the data table described in any one of claims 1 to 11 is implemented.

Images