Data processing method and device of data table, electronic equipment and storage medium

Abstract

The application provides a data processing method and device of a data table, electronic equipment and a storage medium; the method comprises the following steps: receiving service data to be written into a data table, wherein the data table comprises a plurality of data partitions divided by rows, and each data partition comprises a plurality of data buckets divided by rows; writing the service data as a row into a data bucket to be written in a data partition to be written, wherein the data bucket to be written is any one of a plurality of data buckets in the data partition to be written, each data partition comprises a plurality of column clusters divided by columns, each column cluster is used for writing a plurality of associated types of feature data, and the column cluster is physically stored independently; receiving a range query request, wherein the range query request comprises a plurality of types to be queried; reading feature data of each row in each column cluster comprising the plurality of types to be queried from at least one data partition. Through the application, the requirements of efficient storage and high-performance range scanning can be met.

Description

Technical Field

[0001] This application relates to the field of data table technology, and in particular to a data table data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, common massive data storage solutions include HBase, Object Storage, and ClickHouse. HBase is a highly reliable, high-performance, and scalable distributed storage system that supports column-family storage and retrieval of massive datasets. Its underlying storage uses a sparse table data model, resulting in extremely low space consumption for large-scale sparse data storage. However, column families are actually wide-column storage, and row storage is still used internally. In practical business applications, the number of column families is limited, leading to significant bandwidth waste when accessing only a portion of the columns. Furthermore, HBase suffers from poor performance in large-scale scan queries. Object Storage supports massive, unrestricted data with millisecond-level response times, but it doesn't support range scans. Full scans require additional storage components to store the object keys (Object_Key), implemented as single-row queries, resulting in lower performance. ClickHouse is a column-oriented data management system for data analysis, supporting real-time online queries without any data preprocessing, achieving millisecond-level analysis even with hundreds of millions of data points. ClickHouse implements pure column-oriented storage, allowing on-demand access to features with no bandwidth waste. However, in terms of application, ClickHouse is more geared towards real-time analysis and has poor support for data range scanning and single-row data updates (which will generate a large number of small partition files, making the analysis performance worse).

[0003] In other words, there is currently no effective solution in terms of how to balance the needs of efficient storage and high-performance range scanning for massive amounts of data. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for a data table, which can meet the requirements of efficient storage and high-performance range scanning.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a data processing method for a data table, including:

[0007] Receive business data to be written to a data table, wherein the data table includes multiple data partitions divided by rows, and each data partition includes multiple data buckets divided by rows;

[0008] The business data is written as a row to the data bucket to be written in the data partition to be written, wherein the data partition to be written is any one of the multiple data partitions, the data bucket to be written is any one of the multiple data buckets included in the data partition to be written, each data partition includes multiple column families divided by columns, each column family is used to write multiple associated type feature data, and the column families are physically independent.

[0009] Receive a range query request, wherein the range query request includes multiple query types;

[0010] From at least one of the data partitions, read the feature data of each row in each column family that includes the plurality of query types.

[0011] This application provides a data processing apparatus for a data table, including:

[0012] A receiving module is used to receive business data to be written to a data table, wherein the data table includes multiple data partitions divided by rows, and each data partition includes multiple data buckets divided by rows.

[0013] The writing module is used to write the business data as a row to the data bucket to be written in the data partition to be written, wherein the data partition to be written is any one of the multiple data partitions, the data bucket to be written is any one of the multiple data buckets included in the data partition to be written, each data partition includes multiple column families divided by columns, each column family is used to write multiple associated type feature data, and the column families are physically independent;

[0014] The receiving module is further configured to receive a range query request, wherein the range query request includes multiple query types;

[0015] A read module is configured to read feature data of each row in each column family comprising the plurality of query types from at least one of the data partitions.

[0016] This application provides an electronic device, including:

[0017] Memory, used to store executable instructions;

[0018] The processor, when executing executable instructions stored in the memory, implements the data processing method for the data table provided in the embodiments of this application.

[0019] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data processing method for the data table provided in this application.

[0020] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the data processing method for the data table provided in this application.

[0021] The embodiments of this application have the following beneficial effects:

[0022] By dividing the data table into multiple data partitions by rows, further dividing each data partition into multiple data buckets by rows, and further dividing each data partition into multiple column families by columns, each column family is used to write feature data of multiple related types, and each column family is physically independent. In this way, since the features in each column family are stored in a columnar manner, storage efficiency is improved. At the same time, because the column families are physically independent, the column family corresponding to a specific feature can be scanned on demand without reading the entire row of the data table, saving bandwidth. Moreover, multiple related features can be read from the column family at once, thereby improving range scan capability and access performance. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the architecture of the data processing system 100 for the data table provided in this application embodiment;

[0024] Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application;

[0025] Figure 3 This is a flowchart illustrating the data processing method for the data table provided in an embodiment of this application;

[0026] Figure 4 This is a flowchart illustrating the data processing method for the data table provided in an embodiment of this application;

[0027] Figure 5 This is a flowchart illustrating the data processing method for the data table provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the sparse tabular data model provided in the embodiments of this application;

[0029] Figure 7 This is a schematic diagram of the storage model provided in the embodiments of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0034] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0035] 1) Table: A table is the basic unit for organizing data in a database. A table consists of rows and columns.

[0036] 2) Data Partition (Tablet): Also known as a partition, it is the smallest unit of data allocation and is created by dividing a data table into rows. Data partitions are physically stored independently; that is, a data partition cannot be deployed on different servers. When the server hosting a data partition experiences excessive load, the partition can be split according to certain rules.

[0037] 3) Data bucketing (Partition): Simply put, it is a logical unit obtained by dividing a data table by rows. Multiple consecutive data buckets can form a data partition.

[0038] 4) Row key (Sort_Key): Each row in a data table can represent a data object, and each row can be identified by a row key, that is, different rows in the data table can be distinguished by the row key.

[0039] 5) Column Family: This is a collection of columns. Different column families are stored independently. For example, different column families can be stored on different disks, thus enabling the use of different disks. Furthermore, columns within a column family can be sorted lexicographically.

[0040] 6) Write-Ahead Log (WAL): This is a set of techniques used in relational database systems to provide atomicity and durability. In systems using WAL, all data is written to a log file before being committed.

[0041] 7) Log Structure Merge Tree (LSM): This is a hierarchical, ordered, disk-based data structure. An LSM tree consists of a MemTable, an ImmutableMemTable, and a Sorted String Table (SSTable). The MemTable is a data structure in memory used to store recently updated data, organized by key. Because data is temporarily stored in memory, which is not reliable storage and will be lost in the event of a power outage, write-alert (WAL) is typically used to ensure data reliability. When the MemTable reaches a certain size, it is converted into an ImmutableMemTable. The ImmutableMemTable is an intermediate state between the MemTable and the Sorted String Table; write operations are handled by the new MemTable, and the transfer process does not block data updates. The Sorted String Table is an ordered set of key-value pairs, representing the LSM tree on disk. To speed up the reading of the Sorted String Table, key indexes and Bloom filters can be used to expedite key lookups.

[0042] 8) Responding to: used to indicate the conditions or states on which the operation is performed. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0043] This application provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for data tables, which can balance the requirements of efficient storage and high-performance range scanning. The electronic device provided in this application embodiment is described below. The electronic device provided in this application embodiment can be implemented as a server, or it can be implemented collaboratively by a server and terminal devices.

[0044] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the data processing system 100 for the data table provided in this application embodiment. To support applications that balance efficient storage and high-performance range scanning requirements, such as... Figure 1As shown, the data processing system 100 of the data table includes: a server 200 (e.g., a distributed data storage system, a data management system, etc.), a network 300, and a terminal device 400. The network 300 can be a local area network (LAN) or a wide area network (WAN), or a combination of both. The terminal device 400 is a terminal device associated with a user (e.g., a provider of business data). The terminal device 400 can also be called a business-side device. A client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as a client provided by a distributed data storage system, a client provided by a data management system, a browser, etc.

[0045] In some embodiments, server 200 can divide a data table into multiple data partitions by rows, and further divide each data partition into multiple data buckets by rows. Server 200 can then divide each data partition into multiple column families by columns, where each column family can be used to write feature data of multiple associated types, and the column families are physically stored independently. Subsequently, when server 200 receives business data uploaded by a user through terminal device 400, it can write the business data as a row to the data bucket to be written in the data partition. When server 200 subsequently receives a range query request from terminal device 400 carrying multiple query types, it can first extract the multiple query types from the range query request, and then read the feature data of each row in each column family containing the multiple query types from at least one data partition. Finally, server 200 can return the read feature data to terminal device 400, enabling terminal device 400 to display it through the human-computer interaction interface of client 410.

[0046] In other embodiments, the embodiments of this application can also be implemented with the aid of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0047] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0048] Example, Figure 1The server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these. The terminal device 400 and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0049] The structure of the electronic device provided in the embodiments of this application will be described below. Taking the electronic device as a server as an example, see... Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 540, and at least one network interface 520. The various components in the electronic device 500 are coupled together via a bus system 530. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 530 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 530.

[0050] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0051] The memory 540 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 540 may optionally include one or more storage devices physically located away from the processor 510.

[0052] The memory 540 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 540 described in this application embodiment is intended to include any suitable type of memory.

[0053] In some embodiments, memory 540 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0054] Operating system 541 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks;

[0055] The network communication module 542 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0056] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A data processing device 543, which can be software in the form of programs and plug-ins, is shown, comprising the following software modules: a receiving module 5431, a writing module 5432, a reading module 5433, an acquisition module 5434, a partitioning module 5435, a deployment module 5436, a recovery module 5437, and a storage module 5438. These modules are logically connected and can therefore be arbitrarily combined or further split according to the functions implemented. It should be noted that... Figure 2 For ease of explanation, all the above modules are shown at once, but this should not be interpreted as excluding the implementation of the data processing device 543 for the data table, which may include only the receiving module 5431, the writing module 5432, and the reading module 5433. The functions of each module will be described below.

[0057] The data processing method of the data table provided in this application embodiment will be specifically described below with reference to the exemplary application and implementation of the server provided in the embodiments of this application.

[0058] See Figure 3 , Figure 3 This is a flowchart illustrating the data processing method for the data table provided in the embodiments of this application, which will be combined with... Figure 3The steps shown are explained.

[0059] In step 101, business data to be written to the data table is received.

[0060] Here, business data can be various types of data, such as document data, order data, browsing data, etc. This application embodiment does not specifically limit the type of business data. Furthermore, the data table can include multiple data partitions divided by rows, and each data partition can include multiple data buckets divided by rows.

[0061] In some embodiments, before receiving a piece of business data to be written to the data table, the data table can be pre-divided into multiple data partitions evenly (or randomly) by rows, and each data partition can be divided into multiple data buckets evenly (or randomly) by rows.

[0062] For example, taking an order table as an example, when there are 5 different types of orders, the order table can be divided into 5 corresponding data partitions by rows. For instance, this includes data partition 1 for fruit orders (assuming data partition 1 includes rows 1 to 1000), data partition 2 for food orders (assuming data partition 2 includes rows 1001 to 4000), data partition 3 for clothing orders (assuming data partition 3 includes rows 4001 to 5000), data partition 4 for home appliance orders (assuming data partition 4 includes rows 5001 to 6000), and data partition 5 for toy orders (assuming data partition 5 includes rows 6001 to 10000). Furthermore, each data partition can be further divided into multiple data buckets by rows. For example, taking data partition 1 corresponding to fruit orders as an example, data partition 1 can be further divided into data bucket 1 corresponding to local fruits, assuming that data bucket 1 includes rows 1 to 200 of the order table; data bucket 2 corresponding to fruits from other regions, assuming that data bucket 2 includes rows 201 to 300 of the order table; and data bucket 3 corresponding to imported fruits, assuming that data bucket 3 includes rows 301 to 1000 of the order table.

[0063] It should be noted that the number of rows included in different data partitions can be the same or different. For example, all data partitions in a data table can include the same number of rows (i.e., uniform division), or all data partitions can include different numbers of rows (i.e., random division), or some data partitions can include the same number of rows, while others can include different numbers of rows. In addition, the number of rows included in different data buckets can be the same or different. For example, multiple data buckets under the same data partition can include the same number of rows (i.e., uniform division), while data buckets under different data partitions can include different numbers of rows. Of course, different data buckets under the same data partition can also include different numbers of rows (i.e., random division). This application does not specifically limit this.

[0064] In step 102, the business data is written as a row to the data bucket to be written in the data partition to be written.

[0065] Here, the data partition to be written can be any one of multiple data partitions (i.e., the data partition to be written is included in multiple data partitions), and the data bucket to be written can be any one of multiple data buckets included in the data partition to be written (i.e., the data bucket to be written is included in multiple data buckets of the data partition to be written). Each data partition can include multiple column families divided by columns. Each column family can be used to write multiple associated type feature data, and the column families are physically independent, that is, the same column family can only be deployed on one server and cannot be deployed on different servers.

[0066] It's important to clarify that writing business data as a single row to the bucket to be written within the data partition means writing multiple different types of feature data included in the business data into the same row of the bucket. For example, taking document 1 as the business data, multiple different types of feature data of document 1 (such as number of views, number of likes, number of shares, quality score, etc.) can be written into the same row (e.g., the first row in data bucket 1) of the bucket to be written.

[0067] In addition, it should be noted that the feature data of the above-mentioned multiple associated types can refer to feature data with a correlation greater than the correlation threshold. That is to say, when the correlation between multiple feature data is greater than the correlation threshold and continuous read and write are required, the feature data that need to be read and written continuously can be written into the same column family.

[0068] In some embodiments, taking five data partitions as an example, let's assume they are data partition 1, data partition 2, data partition 3, data partition 4, and data partition 5. Data partition 1 includes three data buckets, let's assume they are data bucket 1, data bucket 2, and data bucket 3; data partition 2 includes two data buckets, let's assume they are data bucket 4 and data bucket 5; data partition 3 includes four data buckets, let's assume they are data bucket 6, data bucket 7, data bucket 8, and data bucket 9; data partition 4 includes two data buckets, let's assume they are data bucket 10 and data bucket 11; and data partition 5 includes three data buckets, let's assume they are data bucket 12, data bucket 13, and data bucket 14. Meanwhile, assuming that the data bucket 12 included in the current data partition 5 is in an idle state (i.e. no data has been written), while the data buckets included in other data partitions have all been written with data, the received business data can be written as a row to the data bucket 12 (i.e. the data bucket to be written) in the data partition 5 (i.e. the data partition to be written). For example, multiple different types of feature data included in the business data can be written to the third row of the data bucket 12.

[0069] In other embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the data processing method for the data table provided in the embodiments of this application, as shown below. Figure 4 As shown, Figure 3 Step 102 shown can be achieved through Figure 4 Steps 1021 to 1023 shown are implemented, and will be combined with Figure 4 The steps shown are explained.

[0070] In step 1021, the hash values ​​of the addresses of different servers are mapped to different positions in the hash ring.

[0071] In some embodiments, multiple data partitions can be deployed on different servers. This allows obtaining the addresses of different servers and performing a hash operation on each address to obtain a hash value corresponding to that address. Subsequently, the hash values ​​of the different server addresses can be mapped to different positions in a hash ring. Here, a hash value is a unique and extremely compact numerical representation of a piece of data. For example, taking server 1 as an example, after obtaining the address of server 1, a hash algorithm (such as MD2, MD5, SHA-1, etc.) can be used to calculate the hash value corresponding to the address of server 1.

[0072] For example, consider five data partitions, designated as partitions 1 through 5. These five partitions can be deployed on different servers. For instance, partition 1 can be deployed on server 1, partition 2 on server 2, partition 3 on server 3, partition 4 on server 4, and partition 5 on server 5. Next, the addresses of these five servers, such as Internet Protocol (IP) addresses, can be obtained. These IP addresses can then be hashed, for example using the MD5 algorithm, to obtain the hash values ​​corresponding to each of the five server IP addresses. Finally, the hash values ​​of the IP addresses of these 5 servers can be mapped to different positions in a hash ring that is connected end to end (assuming the hash ring has 8 positions, namely positions 1 to 8). For example, the hash value of the IP address of server 1 (assuming hash value 1) can be mapped to position 1 in the hash ring, the hash value of the IP address of server 2 (assuming hash value 2) can be mapped to position 2 in the hash ring, the hash value of the IP address of server 3 (assuming hash value 3) can be mapped to position 3 in the hash ring, the hash value of the IP address of server 4 (assuming hash value 4) can be mapped to position 4 in the hash ring, and the hash value of the IP address of server 5 (assuming hash value 5) can be mapped to position 5 in the hash ring.

[0073] In step 1022, the address of the business device submitting business data is mapped to the hash ring. Starting from the business device's address in the hash ring, the search proceeds clockwise or counterclockwise, and the server corresponding to the first server mapping position found is taken as the server to be written.

[0074] In some embodiments, following the above, after mapping the hash values ​​of the IP addresses of the five servers to different positions in the hash ring, the IP address of the business device (e.g., a computer) submitting the business data can be further obtained. After obtaining the IP address of the business device, it is also mapped into the hash ring. Starting from the business device's IP address mapping position in the hash ring, a search is performed clockwise or counter-clockwise, and the server corresponding to the first found server mapping position is selected as the server to be written to. For example, taking the business device's IP address mapping position in the hash ring as position 7, and searching clockwise, assuming position 5 is the closest clockwise position to position 7, the server corresponding to position 5 (i.e., server 5) can be selected as the server to be written to.

[0075] It should be noted that after obtaining the IP address of the service device, a hash operation can be performed on the IP address of the service device to obtain the hash value of the IP address of the service device. Then, the hash value of the IP address of the service device can be mapped to the corresponding position in the hash ring. This application embodiment does not specifically limit this.

[0076] In step 1023, at least one data partition deployed on the server to be written is identified as the data partition to be written, and business data is written as a row to the data bucket to be written in the data partition to be written.

[0077] In some embodiments, following the above, after obtaining the server to be written (e.g., server 5) based on the hash ring, at least one data partition deployed on server 5 can be identified as the data partition to be written. Since only data partition 5 is deployed on server 5, data partition 5 can be directly used as the data partition to be written. Then, the business data received in step 101 can be written as a row to the data bucket to be written in data partition 5 (i.e., the data partition to be written). For example, assuming that data partition 5 includes 3 data buckets, namely data bucket 12, data bucket 13 and data bucket 14, where data bucket 13 has no data written, the received business data can be written to data bucket 13 included in data partition 5. In this way, by deploying different data partitions on different servers, load balancing between the servers can be achieved.

[0078] It should be noted that when multiple data partitions are deployed on the server to be written, any one of the data partitions can be used as the data partition to be written. Alternatively, the write rate of each data partition can be determined first, that is, the amount of data that has been written to each data partition relative to the total amount of data that can be written to that data partition. The data partition with the lowest write rate can then be used as the data partition to be written. For example, taking server 1 as the server to be written, there are 3 data partitions deployed on server 1, let's say they are data partition 1, data partition 2 and data partition 3. The current write rate of data partition 1 is 80%, the current write rate of data partition 2 is 90%, and the current write rate of data partition 3 is 85%. In this case, data partition 1 can be used as the data partition to be written, thus avoiding writing a large amount of business data to the same data partition. Of course, data partitions that match the type of business data among multiple data partitions can also be used as data partitions to be written. For example, suppose data partition 1 is used to store data for fruit orders, data partition 2 is used to store data for beverage orders, and data partition 3 is used to store data for clothing orders. If the received business data is order data for clothing A, then data partition 3 can be used as the data partition to be written. This application embodiment does not make specific limitations on this.

[0079] In some embodiments, multiple adjacent column families in a data bucket can share a write-ahead log. When business data intersects with multiple column families to be written in the data bucket, i.e., when business data needs to be written to multiple column families, the following processing can also be performed: multiple instructions for writing business data to multiple column families to be written are stored in the write-ahead log; the business data is written to the data cache of multiple column families to be written; the data cache is synchronized to the server where the data bucket is located so that the server can persist the storage, for example, the server can store the business data on the hard disk to achieve persistent storage; in response to a failure of the server where the data bucket is located, data recovery is performed on multiple column families to be written based on the write-ahead log.

[0080] For example, taking multiple adjacent column clusters 1 to 3 as an example, column clusters 1 to 3 can share a write-ahead log. When business data needs to be written to column clusters 1 to 3, multiple instructions for writing business data in column clusters 1 to 3 can first be stored in the log file, and the business data can be written to the data cache of column clusters 1 to 3. The data cache can also be synchronized to the server where the data bucket resides, so that the server stores the business data on the hard drive. Subsequently, when a failure is detected in the server where the data bucket resides (e.g., a sudden power outage), the log file can be checked to retrieve the previously stored instructions, and data recovery can be performed on column clusters 1 to 3 based on the retrieved instructions.

[0081] In other words, in a system using write-ahead logs, all data modifications must be written to log files before being committed. These log files typically include redo logs and undo logs. Whenever an operation occurs, it can be written to the redo log before the data is changed. This allows the operation to continue after a restart in the event of a power outage or other failure. Furthermore, if some changes cannot be completed halfway through, the undo log can be used to restore the system to its previous state. In other words, the redo log can repair data in the event of a server failure to ensure the durability of transactions, while the undo log ensures the atomicity of transactions.

[0082] In other embodiments, each type of feature of the business data may occupy multiple columns in the column family of the data bucket to be written, wherein one column is used to store the feature value of the feature, and at least two columns are used to store the meta-information of the feature (i.e. information used to describe the feature).

[0083] For example, taking business data as documents, the feature values ​​of document features (such as the document's category, quality, etc.) (such as the number of views, collections, quality rating, etc.) can be written into column 1 of the column family of the data bucket to be written. The metadata of the document features can be written into columns 2 and 3 of the column family. The metadata of the document features refers to the information used to describe the document features. For example, when the document feature is document quality, the corresponding metadata can be the quality evaluation dimension, the number of people participating in the evaluation, etc.

[0084] In some embodiments, the column family of the data to be written to the bucket may also include a secondary index, wherein the secondary index includes multiple related data of features in the business data (e.g., multiple articles published by the same account); the secondary index may occupy one column of multiple columns on its own, or share a column with the feature value.

[0085] For example, taking data bucket 1 as the data to be written, the column family of data bucket 1 can also include secondary indexes. The secondary index can include multiple related data of features in the business data (such as all articles published by a certain account). Furthermore, the secondary index can occupy a row of one of the multiple columns included in the column family to achieve decoupling from the feature value; of course, it can also share a column with the feature value, that is, store the secondary index and the feature value in the same column to improve storage efficiency.

[0086] In other embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the data processing method for the data table provided in the embodiments of this application, as shown below. Figure 5 As shown, after execution Figure 3 After step 102 shown, you can also execute... Figure 5 Step 105 shown will combine Figure 5 The steps shown are explained.

[0087] In step 105, business data is stored in the storage view of the data table in the form of key-value pairs.

[0088] Here, the key in each key-value pair includes the following sort key for each type of feature data in the business data: the data bucket identifier of the data bucket (used to identify the unique corresponding data bucket), the column key of the column, and the row key of the row; the value in the key-value pair is feature data of a type in the business data; and empty data cells in the data table are not assigned key-value pairs.

[0089] In some embodiments, to further improve storage and access bandwidth utilization, after writing business data as a row to the data bucket in the data partition to be written, the business data can be compressed. In the storage view of the data table, the compressed business data can be stored in key-value pairs. Each key-value pair contains the following sorted values ​​for each type of feature data in the business data: the data bucket identifier, the column key, and the row key; the value is a type of feature data in the business data. Furthermore, empty data units are not stored; that is, empty data units in the data table are not assigned key-value pairs. Thus, when storing and accessing large-scale sparse data, the absence of empty data units significantly saves storage space and improves bandwidth utilization.

[0090] In other embodiments, each data unit in the data table can be determined by quaternion information, which includes: the data bucket identifier (e.g., data bucket key) of the data bucket to which the data unit belongs, the column cluster identifier of the column cluster to which the data unit belongs, the row key of the row to which the data unit belongs, and the column key of the column to which the data unit belongs.

[0091] For example, taking data unit 1 as an example, the feature data stored in data unit 1 can be uniquely identified by the four-element information (e.g., a quadruple) of the data bucket identifier of the data bucket in which data unit 1 is located (used to determine which data bucket in the data table data unit 1 is located in), the column family identifier of the column family in which data unit 1 is located (used to determine which column family in the data table data unit 1 is located in), the row key of the row in which data unit 1 is located (used to determine which row in the data bucket data unit 1 is located in), and the column key of the column in which data unit 1 is located (used to determine which column in the column family data unit 1 is located in).

[0092] In step 103, a range query request is received.

[0093] Here, a range query request can include multiple query types. In other words, a range query request is a query request that carries multiple query types.

[0094] In some embodiments, when a user selects multiple types, the multiple types selected by the user can be treated as multiple query types and added to the range query request.

[0095] In step 104, feature data for each row in each column family, which includes multiple query types, is read from at least one data partition.

[0096] In some embodiments, after receiving a range query request, multiple query types can be extracted from the range query request first. Then, feature data of each row in each column cluster containing multiple query types can be read from at least one data partition. That is, by dividing the data table into multiple data partitions by rows, dividing each data partition into multiple data buckets by rows, and dividing each data bucket into multiple column clusters by columns, since each column cluster is stored independently and the feature data in each column cluster is stored column-wise, at least one data partition can be scheduled to perform a range scan by feature search, thereby improving the range scan capability and access performance.

[0097] It should be noted that the number of at least one data partition depends on the query range of the range query request, i.e., the range of the row key to be queried. In other words, the larger the range of the row key to be queried, the larger the number of at least one data partition, and the smaller the range of the row key to be queried, the smaller the number of at least one data partition.

[0098] In other embodiments, the range query request may also include multiple row keys to be searched. After reading the feature data in at least one column family including multiple query types, the following processing may also be performed: retrieving feature data from the rows corresponding to the multiple row keys to be searched from the read feature data.

[0099] For example, when a range query request also carries multiple row keys to be searched, after reading the feature data from at least one column family containing multiple query types, feature data from the rows corresponding to the multiple row keys to be searched can be retrieved from the read feature data. For example, taking multiple row keys to be searched as row key 1 and row key 2, where row key 1 is the row key corresponding to the first row in the data bucket and row key 2 is the row key corresponding to the second row in the data bucket, feature data from the rows corresponding to row key 1 and row key 2 can be retrieved from the read feature data. For example, feature data from the first and second rows in the data bucket can be read to meet more granular search needs.

[0100] It's important to note that when a range query request does not include multiple row keys to be queried, after reading the feature data from at least one column family containing multiple query types, the feature data from all rows within that column family can be retrieved directly from the read feature data. In other words, when a user needs to retrieve feature data from specific rows, they can add the corresponding row keys to the range query request; if the user does not add any row keys to be queried to the range query request, the feature data from all rows will be retrieved. This allows users to flexibly perform searches according to their needs, improving the user experience.

[0101] In some embodiments, the following processes may also be performed: receiving a single query request, wherein the single query request includes a business data identifier to be queried; obtaining the row key to be queried corresponding to the business data identifier to be queried; and reading feature data from the data table located in the row corresponding to the row key to be queried.

[0102] For example, the data table can also be associated with an index table. The index table can include the following mapping relationships: the business data identifier of the business data, and the row key of the row where the business data is located. The row key of the row where the business data is located is a characteristic parameter of the business data. The above-mentioned method of obtaining the row key corresponding to the business data identifier to be queried can be achieved in the following ways: using the business data identifier to be queried as the corresponding row key, or querying the row key corresponding to the business data identifier to be queried from the index table. That is, the business data identifier to be queried (e.g., document identifier) ​​can be directly stored as the row key, so that it can be directly used as the corresponding row key later. Alternatively, a mapping relationship between the business data identifier to be queried and the row key can be established and stored in the index table. In this way, the row key corresponding to the business data identifier to be queried can be queried from the index table later. This embodiment of the application does not specifically limit this approach.

[0103] In other embodiments, multiple data partitions may be deployed on different servers, and the following processing may also be performed: in response to the load parameter of the server where any data partition is located being higher than the load parameter threshold, the data partition is divided into multiple new data partitions evenly (or randomly) by rows; and the multiple new data partitions are deployed to different servers.

[0104] For example, let's consider five data partitions: partition 1, partition 2, partition 3, partition 4, and partition 5. These five partitions can be deployed on different servers. For instance, partition 1 can be deployed on server 1, partition 2 on server 2, partition 3 on server 3, partition 4 on server 4, and partition 5 on server 5. Taking partition 2 as an example, when the load parameters (e.g., CPU utilization, memory utilization) of the server hosting partition 2 (server 2) are detected to be higher than a load parameter threshold—for example, if server 2's memory utilization exceeds the threshold—partition 2 can be evenly divided into multiple new data partitions (each new partition contains fewer rows than partition 2). For example, partition 2 (assuming it includes rows 1000 to 4000 of the table) can be evenly divided into three new data partitions: partition 1 (assuming it includes rows 1000 to 4000 of the table) and partition 5. The system first creates three new data partitions: a first partition containing rows 2000 to 2000, a second partition containing rows 2001 to 3000, and a third partition containing rows 3001 to 4000. These three new data partitions are then deployed on different servers. For example, the first data partition can be deployed on server 1, the second on server 3, and the third on server 4. In this way, when the server containing a certain data partition is overloaded, the data partition can be split into multiple new data partitions and the scheduling can be performed on multiple servers to improve access performance.

[0105] The data processing method for a data table provided in this application divides the data table into multiple data partitions by rows, further divides each data partition into multiple data buckets by rows, and divides each data bucket into multiple column clusters by columns. Each column cluster is used to write feature data of multiple associated types, and each column cluster is physically stored independently. Thus, since the features in each column cluster are stored column-wise, storage efficiency is improved. At the same time, since the column clusters are physically independent, the column clusters corresponding to specific features can be scanned as needed without reading the entire row of the data table, saving bandwidth. Moreover, multiple associated features can be read from the column cluster at once, thereby improving range scanning capability and access performance.

[0106] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0107] This application provides a data processing method for data tables, enabling high-performance, highly scalable, and highly available data storage and access capabilities. Specifically, it improves storage and access bandwidth utilization by employing a sparse table data model, columnar storage + data bucketing (dividing the data table horizontally into multiple data partitions, further dividing each data partition into multiple data buckets, and vertically dividing it into multiple column clusters, storing multiple related features in the same column cluster, with each column cluster stored independently and features stored columnarly within each column cluster, thereby allowing at least one data partition to be scheduled for feature-based range scanning, thus improving range scanning capability and access performance), automatic splitting (splitting a data partition into multiple data partitions, thereby reducing partition request load and improving access performance), and compression (compressing multiple data before writing to reduce data storage volume and cost). Based on this, this application also provides a new table architecture (i.e., a dual-table structure of data table and index table).

[0108] The index table stores the mapping between business data identifiers and row keys. Describing this mapping through the index table allows row keys to include more features of the business data, ensuring that multiple related feature data are physically stored contiguously for enhanced range scan capabilities. Combined with a WAL update mechanism (multiple adjacent column families share a single WAL, and success is only reported when all feature data in different column families has been successfully written, thus improving data write accuracy), custom load balancing (including consistent hashing, segmented consistent hashing, and other load balancing strategies), and a data partitioning mechanism (splitting a data partition into multiple data partitions), it ensures data correctness while providing high-performance, highly scalable range scan capabilities (range scans are based on reading data from multiple features; since column identifiers correspond to features, columnar storage allows for continuous reading of data under the same feature, thus improving range scan efficiency). This makes it an efficient solution for accessing massive amounts of data.

[0109] The data processing method of the data table provided in the embodiments of this application will be described in detail below.

[0110] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of the sparse tabular data model provided in the embodiments of this application, such as... Figure 6As shown, the data is logically stored in a table format, divided horizontally (in the direction corresponding to rows in the table) into several data buckets (partitions) according to a hash-range partitioning strategy. Each data bucket contains multiple rows, and each row is uniquely located using two keys: a data bucket identifier (Partition_Key) and a row key (Sort_Key). Furthermore, the data within a data bucket can be ordered according to the row key, and multiple data buckets are unordered and have no overlap. Multiple consecutive data buckets can form a data partition (Tablet). For example, data buckets 100 and 101 can be combined into a data partition (i.e., data partition _100-101). A data partition is the smallest unit of data scheduling and is physically stored independently. When the partition request load exceeds the load threshold (i.e., the load is too high), for example, when the memory utilization of the server hosting a data partition exceeds the utilization threshold, the data partition can be split according to certain rules. For example, a large data partition can be split into two smaller data partitions and deployed on two separate servers to improve access performance.

[0111] See also Figure 6 In the vertical direction (i.e., the direction corresponding to the columns of the data table), multiple columns (qualifiers) can form a column family, and a data partition can include multiple column families. Furthermore, all column families within the same data partition can share a single Write-Ahead Log (WAL), ensuring the atomicity and integrity of data written to different column families. That is, when writing data to different column families within a data partition, success is only reported when all column families have been successfully written, thus guaranteeing the integrity of the written data. In addition, in the event of a system crash, data can be recovered using the WAL, preventing data loss. Moreover, different column families can use an LSM (Low-Size Module) structure to store data independently. Data that is frequently read and written together (e.g., feature data requiring continuous read / write operations, such as when the correlation between multiple feature data exceeds a correlation threshold) can be stored together in the same column family. Because different column families are physically independent, the column family corresponding to a specific feature can be scanned on demand, without reading the entire row of the data table, saving bandwidth. Furthermore, multiple related feature data can be read from a column family at once, providing higher access performance.

[0112] Furthermore, each data cell in the data table can be uniquely identified by a quadruple of <partition_key, row_key, column family, and column_identifier_ID>. Additionally, due to the use of column families, row keys can be repeated across different column families; for example, row keys within each column family can start counting from 0. Moreover, in the physical view of storage, data is compressed and stored using key-value pairs (KV), and null data cells are not stored. Furthermore, when storing and accessing large-scale sparse data, the use of the LSM data structure ensures that null data cells are not stored, thus significantly saving storage space and improving bandwidth utilization.

[0113] In addition, in terms of storage sorting, business data can be stored in a sorted order according to <data bucket identifier (Partition_Key), column identifier (Qualifier_ID), row key (Sort_Key)>, which allows multiple related feature data to be stored in adjacent columns. When range scanning some features, since multiple related feature data are physically stored contiguously (i.e. stored in the same column family), access bandwidth can be utilized more effectively and higher scan performance can be provided.

[0114] In other embodiments, Figure 6 Based on the sparse tabular data model shown, embodiments of this application also provide, as Figure 7 The storage model shown includes two tables for data storage: a data table and an index table (the index table is used for single-row queries, i.e., querying feature data within a specific row, and cannot be used for range scans). The data table comprises multiple column families, each storing multiple features. Each feature occupies multiple columns within the same column family, storing the following fields: original feature value (occupying one column), feature metadata (occupying multiple columns), and secondary index (Sec_Index). The secondary index is optional; it can occupy a separate column or share a column with the original feature value. The index table stores the mapping information between business data identifiers and row keys (e.g., the mapping relationship between document identifiers and row keys).

[0115] See also Figure 7A piece of business data (such as a document) can include multiple different types of features. Each piece of business data has a unique corresponding business data identifier (Doc_ID) on the business side. Business data can be stored as a row in a data table (that is, multiple features included in the business data are stored in the same row of the data table), and can be uniquely identified by a row key. In most cases, the business data identifier (such as the document identifier) ​​can be stored as the row key. In extended scenarios, the row key can include certain characteristics of the document (such as publication time, account, etc.), ensuring that data with the same characteristics are physically stored contiguously to provide higher range scanning capabilities. Therefore, an index table is needed to store a unique mapping from document identifiers to row keys to ensure that the data can be accessed correctly.

[0116] It's important to note that both index tables and data tables can store index information, but their functions differ. Index tables store unique mappings, providing the ability to access data by document identifier. Secondary indexes in data tables are non-unique mappings, providing the ability to quickly access a batch of data based on a specific feature value, such as accessing all articles published under a particular account, thus accelerating access performance in specific scenarios. Secondary indexes can, in principle, share a column with the original feature value or occupy a separate column. When occupying a separate column, they are decoupled from the original feature value, allowing for more flexible storage of the original feature value and enabling the storage of more complex data structures, such as feature value load testing, serialization, and nested information.

[0117] Furthermore, when business data (such as documents) is accessed, load balancing strategies such as consistent hashing and segmented consistent hashing can be used to write the documents to different data partitions of the data table. Figure 7 Each logic shown is a data bucket, where logic_0 to logic_10 form data partition_1_10. Multiple columns of feature data (such as original feature values ​​and feature metadata) can be written into the data table, and the integrity and correctness of the feature data are guaranteed through the WAL atomic update mechanism. During feature access, the underlying pure columnar sorted storage allows on-demand access during large-scale scans. This means that based on the range of business data provided by the business side (if not specified, it can scan feature data in all rows by default) and the feature range (i.e., multiple features to be scanned), the corresponding column cluster identifier is found through the mapping relationship between features and column cluster identifiers. Feature data in all rows within the column cluster corresponding to the column cluster identifier is then read, and feature data within the row key range is retrieved. Since it does not require reading entire rows, bandwidth utilization is greatly improved compared to row-based storage. During large-scale scans, by simultaneously scanning at least one data partition (the number of data partitions depends on the scan range, i.e., the range of row keys), combined with automatic partition splitting, the load on a single server is reduced, thereby achieving high-performance range scanning requirements.

[0118] In summary, the data processing method for the data table provided in this application, combining data compression, columnar sparse storage, data partitioning, and automatic splitting technologies, reduces storage costs and improves bandwidth utilization. Furthermore, by horizontally scaling service capabilities through data bucketing, it provides high-performance scanning capabilities while reducing the possibility of downtime. Data is stored in the same table, combined with a WAL update mechanism, ensuring data integrity. Considering all these optimizations, in real-world business scenarios, it significantly reduces data storage and access costs.

[0119] The following description continues to illustrate the exemplary structure of the data processing device 543 for the data table provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the data processing device 543 storing the data table in the memory 540 may include: a receiving module 5431, a writing module 5432, and a reading module 5433.

[0120] The receiving module 5431 is used to receive business data to be written to a data table, wherein the data table includes multiple data partitions divided by rows, and each data partition includes multiple data buckets divided by rows; the writing module 5432 is used to write the business data as a row to the data bucket to be written in the data partition to be written, wherein the data partition to be written is any one of the multiple data partitions, and the data bucket to be written is any one of the multiple data buckets included in the data partition to be written, each data partition includes multiple column families divided by columns, each column family is used to write feature data of multiple associated types, and the column families are physically independent; the receiving module 5431 is also used to receive range query requests, wherein the range query request includes multiple query types; the reading module 5433 is used to read the feature data of each row in each column family including multiple query types from at least one data partition.

[0121] In some embodiments, the range query request further includes multiple row keys to be searched; the reading module 5433 is further configured to, after reading feature data in at least one column family including multiple query types, retrieve feature data in rows corresponding to multiple row keys to be searched from the read feature data.

[0122] In some embodiments, multiple data partitions are deployed on different servers; the writing module 5432 is further configured to map the hash values ​​of the addresses of different servers to different positions in a hash ring; map the address of the business device submitting business data to the hash ring, starting from the business device mapping position of the business device address in the hash ring, search in a clockwise or counterclockwise direction, and take the server corresponding to the first server mapping position found as the server to be written; determine at least one data partition deployed on the server to be written as the data partition to be written, and write the business data as a row into the data bucket to be written in the data partition to be written.

[0123] In some embodiments, the receiving module 5431 is further configured to receive a single query request, wherein the single query request includes a business data identifier to be queried; the data processing device 543 of the data table further includes an acquisition module 5434 configured to acquire the row key to be queried corresponding to the business data identifier to be queried; the reading module 5433 is further configured to read feature data located in the row corresponding to the row key to be queried from the data table.

[0124] In some embodiments, the data table is associated with an index table, which includes the following mapping relationships: business data identifier of the business data, and row key of the row where the business data is located; wherein, the row key of the row where the business data is located is the characteristic parameter of the business data; the acquisition module 5434 is further configured to use the business data identifier to be queried as the corresponding row key to be queried, or to query the row key to be queried corresponding to the business data identifier to be queried from the index table.

[0125] In some embodiments, multiple data partitions are deployed on different servers; the data processing device 543 of the data table further includes a partitioning module 5435 and a deployment module 5436, wherein the partitioning module 5435 is used to partition any data partition into multiple new data partitions by row in response to the load parameter of the server where any data partition is located being higher than the load parameter threshold; the deployment module 5436 is used to deploy the multiple new data partitions to different servers.

[0126] In some embodiments, multiple adjacent column clusters in a data bucket share a write-ahead log. When business data intersects with multiple column clusters to be written in the data bucket, the write module 5432 is further configured to store multiple instructions for writing business data in the multiple column clusters to be written into the write-ahead log, write the business data into the data cache of the multiple column clusters to be written, and synchronize the data cache to the server where the data bucket is located so that the server can persist the storage. The data processing device 543 of the data table also includes a recovery module 5437, which is configured to perform data recovery on the multiple column clusters to be written based on the write-ahead log in response to a failure of the server where the data bucket is located.

[0127] In some embodiments, the data processing device 543 of the data table further includes a storage module 5438, which is used to store the business data in the storage view of the data table in the manner of key-value pairs after the writing module 5432 writes the business data as a row to the data bucket to be written in the data partition to be written. The key in each key-value pair includes the following sort key for each type of feature data in the business data: the data bucket identifier of the data bucket, the column key of the column, and the row key of the row; the value in the key-value pair is feature data of a type in the business data; empty data cells in the data table are not assigned key-value pairs.

[0128] In some embodiments, each type of feature of the business data occupies multiple columns in the column family of the data bucket to be written, wherein one column is used to store the feature value of the feature, and at least two columns are used to store the meta-information of the feature.

[0129] In some embodiments, the column family of the data to be written to the bucket also includes a secondary index, which includes multiple related data of features in the business data; the secondary index occupies one column of multiple columns alone, or shares a column with the feature values.

[0130] In some embodiments, each data cell in the data table is determined by quaternion information, which includes: the data bucket identifier of the data bucket in which the data cell belongs, the column family identifier of the column family in which the data cell belongs, the row key of the row in which the data cell belongs, and the column key of the column in which the data cell belongs.

[0131] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore it will not be repeated. For technical details not covered in the data processing apparatus for the data table provided in this application embodiment, please refer to... Figure 3 , Figure 4 ,or Figure 5 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0132] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to perform the data processing method for the data table described in this application embodiment.

[0133] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the data processing method for the data table provided in this application. For example, ... Figure 3 , Figure 4 ,or Figure 5 The data processing method shown in the data table.

[0134] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0135] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0136] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0137] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method for a data table, characterized in that, The method includes: Receive business data to be written to a data table, wherein the data table includes multiple data partitions divided by rows, and each data partition includes multiple data buckets divided by rows; The business data is written as a row to the data bucket to be written in the data partition to be written. The data partition to be written is any one of the plurality of data partitions, and the data bucket to be written is any one of the plurality of data buckets included in the data partition to be written. Each data partition includes multiple column families divided by columns. Each column family is used to write multiple associated type feature data, and each column family is physically independent. The feature data in each column family is stored in columnar format. The feature data of the multiple associated types are feature data with a correlation greater than a correlation threshold. When the correlation between multiple feature data is greater than the correlation threshold, and the multiple feature data are read and written consecutively, the multiple feature data are treated as feature data of the multiple associated types and written into the same column family. In the storage view of the data table, the business data is stored in key-value pairs. The key in each key-value pair includes the following sort key for each type of feature data in the business data: the data bucket identifier of the data bucket, the column key of the column, and the row key of the row. The value in the key-value pair is a type of feature data in the business data. Empty data cells in the data table are not assigned key-value pairs. Receive a range query request, wherein the range query request includes multiple query types and multiple search row keys; From at least one of the data partitions, feature data of each row in each column family including the plurality of query types is read, and feature data of the rows corresponding to the plurality of search row keys is retrieved from the read feature data.

2. The method according to claim 1, characterized in that, The multiple data partitions are deployed on different servers; The step of writing the business data as a row to the data bucket to be written in the data partition includes: Map the hash values ​​of the addresses of the different servers to different positions in the hash ring; The address of the business device that submits the business data is mapped to the hash ring. Starting from the business device address in the hash ring, the search proceeds clockwise or counterclockwise, and the server corresponding to the first server mapping position found is taken as the server to be written. At least one data partition deployed on the server to be written is identified as the data partition to be written, and the business data is written as a row to the data bucket to be written in the data partition to be written.

3. The method according to claim 1, characterized in that, The method further includes: Receive a single query request, wherein the single query request includes the identifier of the business data to be queried; Obtain the row key corresponding to the business data identifier to be queried; Read the feature data from the row corresponding to the row key to be queried from the data table.

4. The method according to claim 3, characterized in that, The data table is associated with an index table, which includes the following mapping relationships: the business data identifier of the business data, and the row key of the row where the business data is located; wherein, the row key of the row where the business data is located is the characteristic parameter of the business data; The step of obtaining the row key corresponding to the business data identifier to be queried includes: Alternatively, the identifier of the business data to be queried can be used as the corresponding row key, or the row key corresponding to the identifier of the business data to be queried can be queried from the index table.

5. The method according to any one of claims 1 to 4, characterized in that, The multiple data partitions are deployed on different servers; The method further includes: In response to the fact that the load parameter of the server where any of the data partitions resides is higher than the load parameter threshold, the data partition is divided into multiple new data partitions by row; The multiple new data partitions are deployed to the different servers.

6. The method according to any one of claims 1 to 4, characterized in that, In the data bucket, multiple adjacent column families share a single write-ahead log; When the business data intersects with multiple column families to be written in the data bucket to be written, the method further includes: Multiple instructions for writing the business data into the multiple column families to be written are stored in the write-ahead log. The business data is written into the data cache of the multiple column families to be written. The data cache is synchronized to the server where the data bucket is located so that the server can persist the data. In response to a failure of the server where the data bucket is located, data recovery is performed on the multiple column clusters to be written based on the write-ahead log.

7. The method according to any one of claims 1 to 4, characterized in that, Each type of feature of the business data occupies multiple columns in the column family of the data bucket to be written, wherein one column is used to store the feature value of the feature, and at least two columns are used to store the meta-information of the feature.

8. The method according to claim 7, characterized in that, The column family of the data bucket to be written also includes a secondary index, which includes multiple related data of features in the business data; the secondary index occupies one column of multiple columns alone, or shares a column with the feature value.

9. The method according to any one of claims 1 to 4, characterized in that, Each data unit in the data table is determined by quaternion information, which includes: The data bucket identifier of the data bucket to which the data unit belongs, the column cluster identifier of the column cluster to which the data unit belongs, the row key of the row to which the data unit belongs, and the column key of the column to which the data unit belongs.

10. A data processing device for a data table, characterized in that, The device includes: A receiving module is used to receive business data to be written to a data table, wherein the data table includes multiple data partitions divided by rows, and each data partition includes multiple data buckets divided by rows. The write module is used to write the business data as a row to a data bucket within a data partition to be written. The data partition to be written is any one of the plurality of data partitions, and the data bucket to be written is any one of the plurality of data buckets included in the data partition to be written. Each data partition includes multiple column families divided by columns. Each column family is used to write multiple associated type feature data, and each column family is physically independent. The feature data in each column family is stored in columnar format. The feature data of the multiple associated types are feature data with a correlation greater than a correlation threshold. When the correlation between multiple feature data is greater than the correlation threshold, and the multiple feature data are read and written consecutively, the multiple feature data are treated as feature data of the multiple associated types and written into the same column family. A storage module is used to store the business data in a key-value pair format in the storage view of the data table. Each key-value pair contains the following sort key for each type of feature data in the business data: the data bucket identifier of the data bucket, the column key of the column, and the row key. The value in each key-value pair is a type of feature data in the business data. Empty data cells in the data table are not assigned key-value pairs. The receiving module is further configured to receive a range query request, wherein the range query request includes multiple query types and multiple search row keys; The reading module is configured to read feature data of each row in each column cluster including the plurality of query types from at least one of the data partitions, and retrieve feature data of the rows corresponding to the plurality of search row keys from the read feature data.

11. The apparatus according to claim 10, characterized in that, In the data bucket, multiple adjacent column clusters share a write-ahead log. When the business data intersects with multiple column clusters to be written in the data bucket, the writing module is further configured to store multiple instructions for writing the business data in the multiple column clusters to be written into the write-ahead log, write the business data into the data cache of the multiple column clusters to be written, and synchronize the data cache to the server where the data bucket is located, so that the server can persistently store the data. The data processing device of the data table also includes a recovery module, which is configured to perform data recovery on the multiple column clusters to be written based on the write-ahead log in response to a failure of the server where the data bucket is located.

12. The apparatus according to claim 10, characterized in that, Each type of feature of the business data occupies multiple columns in the column family of the data bucket to be written, wherein one column is used to store the feature value of the feature, and at least two columns are used to store the meta-information of the feature.

13. The apparatus according to claim 12, characterized in that, The column family of the data bucket to be written also includes a secondary index, which includes multiple related data of features in the business data; the secondary index occupies one column of multiple columns alone, or shares a column with the feature value.

14. The apparatus according to any one of claims 10 to 13, characterized in that, Each data unit in the data table is determined by four-element information, which includes: the data bucket identifier of the data bucket to which the data unit belongs, the column cluster identifier of the column cluster to which the data unit belongs, the row key of the row to which the data unit belongs, and the column key of the column to which the data unit belongs.

15. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the data processing method of the data table according to any one of claims 1 to 9.

16. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the data processing method of the data table according to any one of claims 1 to 9.

17. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by the processor, the data processing method of the data table according to any one of claims 1 to 9 is implemented.

Images