Data writing method, data reading method and apparatus

Abstract

The application discloses a data writing method, a data reading method and devices. The application solves the problem that a distributed database uses a built-in table engine and a concurrent conflict occurs when facing a super large scale label data writing and reading. The data writing method comprises the following steps: determining a target node and a target label table corresponding to to-be-written data according to a user identifier and a label identifier of the to-be-written data; generating a writing task corresponding to each label data respectively; and distributing each writing task to each thread, so that each thread writes the label data into the target label table respectively. The application improves the writing and reading efficiency, shortens the writing and reading time, and avoids the influence of writing and reading timeout on business development.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to data writing methods, data reading methods, and apparatus. Background Technology

[0002] In existing technologies, to address the low performance of point-to-point queries in distributed databases, a built-in table engine is typically used to improve query efficiency. Due to the architecture of distributed databases, when business data is written to the distributed database, the application layer needs to perform data sharding and master-slave replication. Similarly, when reading tagged data from the distributed database, the application layer also needs to read it according to the sharding rules.

[0003] In the process of realizing this invention, the inventors discovered that the prior art has at least the following problems:

[0004] When faced with writing and reading extremely large amounts of tag data, distributed databases may experience concurrency conflicts, resulting in problems such as slow write speeds, write timeouts, and slow read response times. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a data writing method, a data reading method, and an apparatus, with the aim of improving the speed of writing and reading tag data in a distributed database, and avoiding slow writing and reading speeds that cause timeouts and affect business operations.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a data writing method is provided, comprising:

[0007] Based on the user identifier and tag identifier of the data to be written, the target node and target tag table corresponding to the data to be written are determined; wherein, the data to be written includes tag data under at least one tag, and the target node stores a tag table that corresponds one-to-one with each tag;

[0008] Write tasks corresponding to each of the aforementioned tag data are generated separately, and each write task is assigned to a different thread so that each thread writes the tag data to the target tag table. The threads execute each write task in parallel, and write tasks corresponding to tag data that are identical in both the target node and the target tag table are assigned to the same thread.

[0009] Furthermore, based on the user identifier and tag identifier of the data to be written, the target node and target tag table corresponding to the data to be written are determined, including:

[0010] The target node corresponding to the data to be written is determined based on the user identifier of the data to be written.

[0011] For each piece of tag data, the target tag table corresponding to the tag data is determined based on the tag identifier corresponding to the tag data.

[0012] Furthermore, based on the user identifier of the data to be written, the target node corresponding to the data to be written is determined, including:

[0013] Obtain the user serial number corresponding to the user identifier of the data to be written, wherein the user serial number is obtained by sorting each user identifier according to the first sorting rule;

[0014] Based on the total number of nodes and the user sequence number, and by calculating the node sequence number according to the horizontal sharding algorithm, the node corresponding to the node sequence number is determined as the target node corresponding to the data to be written.

[0015] Furthermore, based on the user identifier and tag identifier of the data to be written, the target node and target tag table corresponding to the data to be written are determined, including:

[0016] Obtain the user serial number corresponding to the user identifier of the data to be written, wherein the user serial number is obtained by sorting each user identifier according to the first sorting rule;

[0017] Obtain the tag number corresponding to the tag identifier of the data tag, wherein the tag number is obtained by sorting each tag identifier according to the second sorting rule;

[0018] Based on the total number of nodes, the user serial number, and the tag serial number, the target tag table serial number is calculated, thereby determining the target node and target tag table corresponding to the tag data.

[0019] Furthermore, it also includes: constructing a tag matrix with each tag and each node as rows and columns respectively, wherein each element in the tag matrix represents a tag table number, and each tag table number is obtained according to a third sorting rule;

[0020] Based on the total number of nodes, the user serial number, and the tag serial number, the target tag table serial number is calculated to determine the target node and target tag table corresponding to the tag data, including:

[0021] Subtract one from the tag number and multiply by the total number of nodes to obtain the first encoding value. Subtract one from the user number and divide by the total number of nodes to obtain the second encoding value. Add the first encoding value, the second encoding value and one to obtain the target tag table number.

[0022] The target node and target label table corresponding to the label data are determined based on the target label table sequence number.

[0023] According to a second aspect of the present invention, a data reading apparatus is provided for reading data written using the data writing method, comprising:

[0024] The user identifier and tag identifier are parsed from the data read request, and the target tag table is determined based on the user identifier and the tag identifier;

[0025] Read the target label data from the target label table.

[0026] According to a third aspect of the present invention, a data writing apparatus is provided, characterized in that it comprises:

[0027] The target determination module is used to determine the target node and target tag table corresponding to the data to be written based on the user identifier and tag identifier of the data to be written; wherein, the data to be written includes tag data under at least one tag, and the target node stores a tag table that corresponds one-to-one with each tag;

[0028] The data writing module is used to generate writing tasks corresponding to each of the tag data, and to assign each writing task to a thread, so that each thread writes the tag data to the target tag table respectively; wherein, each thread executes each writing task in parallel, and the writing tasks corresponding to tag data that are the same in both the target node and the target tag table are assigned to the same thread.

[0029] According to a fourth aspect of the present invention, an electronic processing apparatus is provided, comprising:

[0030] One or more processors; and,

[0031] A memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, cause the one or more processors to implement the method provided in the first aspect of the present invention.

[0033] According to a fifth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0034] According to a sixth aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided in the first aspect of the present invention.

[0035] One embodiment of the invention has the following advantages or beneficial effects:

[0036] Because the distributed database stores a tag table corresponding to each tag in each node, tag data can be written to each tag table in parallel on each target node during writing, and parallel processing is also convenient during reading. Therefore, it overcomes the technical problem of concurrency conflicts that occur when using a built-in table engine in existing distributed databases to handle extremely large-scale tag data writing and reading. This invention improves writing and reading efficiency, shortens writing and reading time, and avoids writing and reading timeouts that could affect business operations by writing tag data to each tag table in parallel on each target node. Attached Figure Description

[0037] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0038] Figure 1 This is a schematic diagram of the main flow of the data writing method according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram illustrating the principle of the data to be written and the distributed storage according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram illustrating the writing principle of data to be written according to an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the main flow of a data writing method according to another embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the main flow of a data writing method according to another embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the tag matrix in a data writing method according to another embodiment of the present invention;

[0044] Figure 7 This is a schematic diagram of the main flow of the data reading method according to an embodiment of the present invention;

[0045] Figure 8 This is a schematic diagram illustrating the principle of the data writing method and the data reading method according to an embodiment of the present invention;

[0046] Figure 9 This is a schematic diagram of the main modules of a data storage device according to an embodiment of the present invention;

[0047] Figure 10 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0048] Figure 11This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0049] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0050] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0051] Figure 1 This is a schematic diagram illustrating the main flow of a data writing method according to an embodiment of the present invention. Figure 1 As shown, the data writing method in this embodiment of the invention includes the following steps S101 and S102.

[0052] Step S101: Determine the target node and target tag table corresponding to the data to be written based on the user identifier and tag identifier of the data to be written; wherein, the data to be written includes tag data under at least one tag, and the target node stores a tag table that corresponds one-to-one with each tag.

[0053] Specifically, in this embodiment and some other embodiments of the present invention, determining the target node and target tag table corresponding to the data to be written based on the user identifier and tag identifier of the data to be written includes: determining the target node corresponding to the data to be written based on the user identifier of the data to be written; and for each tag data, determining the target tag table corresponding to the tag data based on the tag identifier corresponding to the tag data.

[0054] Understandably, the data to be written includes various types of tagged data, such as personal health data, personal credit data, static product data, and corporate credit data. This data is used to build characteristic models of specific individuals or groups in order to better understand, analyze, and serve those individuals or groups. Figure 2 A schematic diagram illustrating the data to be written and the principle of distributed storage in this embodiment is provided. For example... Figure 2As shown, the data to be written in this embodiment is user profile data, including tag data of 90 tags for 100 million users. Figure 2 The user profile data in this embodiment is presented in tabular form, but the data to be written in this embodiment is not limited to tabular form. The use of tabular form is for illustration only and not for limitation. Figure 2 The document specifically displays tag data for four users, identified as East7, South5, West3, and Norht9. The four tags shown are age, occupation, education level, and hobbies, labeled Tag 1, Tag 2, Tag 3, and Tag 90, respectively. Figure 2 It is known that each user's user profile data includes 90 tag data that correspond one-to-one with each tag identifier, and the tag data corresponding to each tag identifier includes 100 million users and the tag data corresponding to that tag identifier.

[0055] Understandably, to understand, analyze, and serve specific user groups, it is often necessary to collect extremely large amounts of tag data. Due to the massive volume of data, tag data is typically stored and managed using distributed storage methods. For example... Figure 2 As shown, there are 4 nodes in the distributed storage. Each node stores 90 tag tables that correspond one-to-one with each tag of the user profile data. Each tag table stores the tag data of a corresponding tag. The tag data of each user profile data is stored in the 90 tag tables of one node. Each node stores user profile data of 25 million people.

[0056] Understandably, determining the target node and target tag table corresponding to the data to be written is based on various preset mapping rules between the data to be written and the node and tag tables. For example, Figure 2 The users shown are distributed across four regions: East, South, West, and North. Mapping rules are set according to these regions (users with "East" in their identifier map to node 1, "South" to node 2, "West" to node 3, and "North" to node 4). User profile data for each region is stored in [location missing]. Figure 2 The corresponding nodes of the four nodes shown each have their respective tag data stored in the tag table corresponding to the tag identifier of each user profile. According to the preset mapping rule, the target node for the data to be written for user identifier "West3" is node 3, the target table for the tag data "44" of tag 1 (age) is tag table 1 on node 3, ..., and the target tag table for the tag data "skiing" of tag 90 (hobby) is tag table 90 on node 3.

[0057] Step S102: Generate write tasks corresponding to each of the tag data, and assign each write task to a thread, so that each thread writes the tag data to the target tag table respectively; wherein, each thread executes each write task in parallel, and write tasks corresponding to tag data that are the same in both the target node and the target tag table are assigned to the same thread.

[0058] Understandably, after step S101 is completed in this embodiment, the target node and target tag table for each piece of data to be written are determined, that is, the target node and current tag table for each tag data of each piece of data to be written are determined. Writing a piece of data to be written to a target node means writing each tag data of the data to be written to each target tag table of the target node. Each tag data corresponds to a write task, and the write tasks corresponding to the same tag data are assigned to the same thread when the target node and target tag table are the same. Figure 3 This is a schematic diagram illustrating the writing principle of the data to be written (user profile data) according to this embodiment. For example... Figure 3 As shown, specifically, for each user profile data with target node 2, the corresponding write tasks for tag data of tag 1 are write task a1, write task a2, ..., which are assigned to thread 1; the corresponding write tasks for tag data of tag 2 are write task b1, write task b2, ..., which are assigned to thread 2; ...; the corresponding write tasks for tag data of tag 90 are write task x1, write task x2, ..., which are assigned to thread 90. Thread 1, Thread 2, ..., and Thread 90, a total of 90 threads, execute in parallel. Thread 1 writes the tag data (tag data a1, tag data a2, ...) of tag 1 for the 25 million user profile data of target node 2 to tag table 1; Thread 2 writes the tag data (tag data b1, tag data b2, ...) of tag 2 for the 25 million user profile data of target node 2 to tag table 2; ..., Thread 90 writes the tag data (tag data x1, tag data x2, ...) of tag 90 for the 25 million user profile data of target node 2 to tag table 90. By executing these 90 threads in parallel, the 90 tag profile data for each of the 25 million users on target node 2 are written to 90 target data tables on the target node (node ​​2). This overcomes the concurrency conflict problem that occurs when using a built-in table engine in a distributed database to write ultra-large-scale tag data, improves writing efficiency, shortens writing time, and avoids write timeouts affecting business operations.

[0059] Optionally, the main flow of the data writing method in other embodiments of the present invention is as follows: Figure 4 As shown, it includes steps S401, S402 and S403.

[0060] Step S401: Determine the target node corresponding to the data to be written based on the user identifier of the data to be written.

[0061] Step S402: For each tag data, determine the target tag table corresponding to the tag data based on the tag identifier corresponding to the tag data.

[0062] Step S403: Generate write tasks corresponding to each of the tag data, and assign each write task to a thread, so that each thread writes the tag data to the target tag table respectively; wherein, each thread executes each write task in parallel, and write tasks corresponding to tag data that are the same in both the target node and the target tag table are assigned to the same thread.

[0063] The data to be written includes tag data under at least one tag, and the target node stores a tag table that corresponds one-to-one with each tag. In step S401, determining the target node corresponding to the data to be written based on the user identifier includes:

[0064] Obtain the user serial number corresponding to the user identifier of the data to be written, wherein the user serial number is obtained by sorting each user identifier according to the first sorting rule;

[0065] Based on the total number of nodes and the user sequence number, and by calculating the node sequence number according to the horizontal sharding algorithm, the node corresponding to the node sequence number is determined as the target node corresponding to the data to be written.

[0066] Furthermore, in these and other embodiments of the present invention, the node sequence number is calculated according to the total number of nodes and the user sequence number, and the node corresponding to the node sequence number is determined as the target node corresponding to the data to be written, including: dividing the data sequence number by the number of nodes and obtaining the remainder as the node sequence number; or, dividing the hash value of the data sequence number by the number of nodes and obtaining the remainder as the node sequence number.

[0067] Optionally, the main flow of the data writing method in other embodiments of the present invention is as follows: Figure 5 As shown, it includes steps S501, S502, S503 and S504.

[0068] Step S501: Obtain the user serial number corresponding to the user identifier of the data to be written. The user serial number is obtained by sorting each user identifier according to the first sorting rule.

[0069] Step S502: Obtain the tag number corresponding to the tag identifier of the data tag. The tag number is obtained by sorting each tag identifier according to the second sorting rule.

[0070] Step S503: Calculate the target tag table number based on the total number of nodes, the user number, and the tag number, thereby determining the target node and target tag table corresponding to the tag data.

[0071] Step S504: Generate write tasks corresponding to each of the tag data, and assign each write task to a thread, so that each thread writes the tag data to the target tag table respectively; wherein, each thread executes each write task in parallel, and write tasks corresponding to tag data that are the same in both the target node and the target tag table are assigned to the same thread.

[0072] Understandably, the terms "first" and "second" are used for purpose only in relation to sorting rules and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Features defined by "first" and "second" in this invention may explicitly or implicitly include at least one of those features. Thus, the first sorting rule and the second sorting rule, etc., represent the target of the sorting rule, not a limitation on the sorting rule.

[0073] Furthermore, in these embodiments of the present invention, the method further includes: constructing a tag matrix using each tag and each node as rows and columns, respectively, wherein each element in the tag matrix represents a tag table sequence number, and each tag table sequence number is obtained according to a third sorting rule, such as... Figure 6 As shown, nodes m1, m2, and m3 represent nodes with serial numbers 1, 2, and 3, respectively; labels n1, n2, ..., n90 represent labels with serial numbers 1, 2, ..., 90, respectively; and label tables P1, P2, ..., P270 represent label tables with serial numbers 1, 2, ..., 90, respectively. For the label data to be written, the target node and target label table can be quickly obtained based on the mapping relationship between the target label table serial number and the target node and target label table in the label matrix.

[0074] Specifically, in these embodiments and other embodiments of the present invention, the target tag table number is calculated based on the total number of nodes, the user serial number, and the tag serial number, thereby determining the target node and target tag table corresponding to the tag data, including:

[0075] The tag number minus one and multiplied by the total number of nodes yields a first encoded value; the user number minus one and divided by the total number of nodes yields a second encoded value, specifically, the remainder obtained by subtracting one from the user number and taking the modulo of the total number of nodes is used as the second encoded value; the first encoded value, the second encoded value, and one are added together to obtain the target tag table number; the target node and target tag table corresponding to the tag data are determined based on the target tag table number. That is, the target node and target tag table corresponding to the tag data are determined based on the mapping relationship between the target tag table number and the target node and target tag table in the tag matrix.

[0076] Optionally, in some embodiments of the present invention, for scenarios such as online analysis and processing databases and key-value pair distributed storage, the fields of the tag table may include user identifier, tag identifier, tag data, and tag data write time, so as to adapt to big data analysis and processing such as online analysis and processing and key-value pair processing.

[0077] According to a second aspect of the present invention, a data reading method is provided, such as... Figure 7 As shown, the method for reading data written using the data writing method includes steps S701 and S702.

[0078] Step S701: Parse the user identifier and tag identifier from the data read request, and determine the target tag table based on the user identifier and the tag identifier.

[0079] It is understood that, in this embodiment of the invention, the same method as the data writing method in this embodiment of the invention can be used to determine the user identifier and tag identifier of the data to be read requested by the data read request.

[0080] Step S702: Read target tag data from the target tag table.

[0081] It is understood that in this embodiment and some other embodiments of the present invention, after determining the target node and target tag identifier of the data to be read, the target tag data of the data to be read can be read by executing the corresponding structured query instruction at the target node.

[0082] It is understood that in this embodiment and some other embodiments of the present invention, step S702 includes: for the same target node, generating a structured query instruction, such as an SQL statement, to read the data to be read on the target node based on the tag identifier of each tag data of the data to be read.

[0083] Furthermore, in this embodiment and some other embodiments of the present invention, step S702 includes:

[0084] In response to identical tag identifiers in the tag data, a single tag table corresponding to the tag identifier is determined on the target node, and a single-table structured query statement is generated as a structured query instruction to read the data to be read on the target node.

[0085] In response to different tag identifiers in the tag data, for a group of tag data with the same tag identifier, a single tag table corresponding to the tag identifier is determined on the target node, and a single-table structured query statement for the group is generated. The single-table structured query statements of each group are concatenated to obtain a multi-table structured query statement, which serves as a structured query instruction for reading the data to be read on the target node.

[0086] Understandably, since each node in the distributed database stores a tag table that corresponds one-to-one with each tag, tag data can be written to each tag table in parallel on each target node when it is written, and it can also be processed in parallel when it is read. This overcomes the technical problem of concurrency conflicts that occur when using a built-in table engine in the distributed database to read ultra-large-scale tag data in the existing technology, improves reading efficiency, shortens reading time, and avoids reading timeouts that affect business operations.

[0087] Furthermore, Figure 8 This is a schematic diagram illustrating the principle of the data writing method and data reading method in one embodiment of the present invention. Figure 8 As shown, in this embodiment, the data writing method and data reading method are applied to the ClickHouse distributed storage database using the built-in RocksDB table engine, and the application layer adopts the Spark distributed computing framework.

[0088] ClickHouse is a high-performance, stable, and easy-to-use open-source columnar database designed for Online Analytical Processing (OLAP). Through columnar storage and a distributed architecture, it enables efficient analysis and real-time querying of big data, making it particularly suitable for scenarios requiring real-time big data analysis, such as user behavior analysis and traffic statistics. It is an important tool for enterprises to conduct data mining and decision support. The data to be written in this embodiment is also user profile data, including tag data for 90 tags representing 150 million users.

[0089] ClickHouse incorporates the RocksDB table engine. RocksDB is a high-performance, scalable key-value table engine for handling table storage, retrieval, and management. It employs an LSM tree data structure, supports fast storage and efficient range queries, and is suitable for various production environments and workloads. RocksDB offers flexible configuration options and rich features, such as snapshots and transactions, enabling it to handle a variety of complex application scenarios. Furthermore, key-value data is a simple data storage model where the key serves as a unique identifier, and the value stores the actual data content. This model organizes, indexes, and stores data in key-value pairs, featuring simple structure, efficient read / write operations, and easy scalability. It is particularly suitable for scenarios with less demanding data structure requirements, such as caching, data storage, and counters.

[0090] Spark is a fast, general-purpose, and scalable distributed computing framework designed to simplify large-scale data processing tasks. A Spark Executor is a key component of the Spark framework; it is a worker process running on cluster nodes. A Spark Executor is responsible for executing application tasks (Spark tasks), processing data computations and operations, and caching data and managing memory. Each Spark Executor is allocated a certain amount of CPU and memory resources and can execute multiple tasks, supporting the operation of the entire Spark application.

[0091] like Figure 8 As shown, in this embodiment, the ClickHouse distributed storage database has 3 groups of 6 nodes. Each group includes a master node and 2 slave nodes, with the master and slave nodes serving as backups for each other. When writing data, it is written to the master node first and then to the slave node; when reading data, a load balancing algorithm, such as a random algorithm or a round-robin algorithm, is used to select one of the master and slave nodes for reading. In this embodiment,

[0092] like Figure 8 As shown, the data writing method in this embodiment includes:

[0093] Step S811: Obtain the user sequence number corresponding to the user identifier of the data to be written. The user sequence number is obtained by sorting the user identifiers according to the first sorting rule. Specifically, the corresponding user sequence number is obtained from the data to be written in the Spark job.

[0094] Step S812 involves obtaining the tag number corresponding to the tag identifier of the data tag. The tag number is obtained by sorting the tag identifiers according to the second sorting rule. Specifically, the corresponding tag number is obtained from the data to be written in the Spark job.

[0095] Step S813: Based on the total number of nodes, the user sequence number, and the tag sequence number, calculate the target tag table sequence number to determine the target node and target tag table corresponding to the tag data. Specifically, this embodiment employs the following... Figure 6 The label matrix shown is implemented using the Repartition() operator of the Spark distributed computing framework. Target label table sequence number. The calculation method is as follows:

[0096]

[0097] in, Indicates the label number. This indicates the number of nodes (in this embodiment, the master and slave nodes are considered as the same node because they are backups of each other, so the value is 3). It is the user serial number, and [*] indicates taking the modulo of *.

[0098] Step S814: Generate write tasks corresponding to each of the tag data, and assign each write task to a thread, so that each thread writes the tag data to the target tag table; wherein, the threads execute the write tasks in parallel, and write tasks corresponding to tag data where the target node and the target tag table are the same are assigned to the same thread. Specifically, each thread is... Figure 8 The Spark task1, Spark task2, ..., Spark task90 shown are executed in parallel by the corresponding Spark Executors running on the three nodes of the distributed storage. Figure 8 The image only shows Spark tasks executed in parallel by Spark Executor2 corresponding to node 2; the same applies to other nodes.

[0099] Furthermore, such as Figure 8 As shown, the data reading method in this embodiment includes:

[0100] Step S821: Parse the user identifier and tag identifier from the data read request, and determine the target tag table based on the user identifier and the tag identifier.

[0101] Step S822: Read target tag data from the target tag table. Specifically, in this embodiment, this includes:

[0102] In response to identical tag identifiers in the tag data, a single tag table corresponding to the tag identifier is determined on the target node, and a single-table SQL statement is generated. A node is selected from the master and slave nodes of the target node according to the load balancing algorithm, and data is read on the selected node using the single-table SQL statement.

[0103] In response to different tag identifiers in the tag data, for a group of tag data with the same tag identifier, a single tag table corresponding to the tag identifier is determined on the target node, and a single-table SQL statement for the group is generated. The single-table SQL statements of each group are concatenated to obtain a concatenated SQL statement. A node is selected from the master node and slave node of the target node according to the load balancing algorithm, and the data is read on the selected node using the concatenated SQL statement.

[0104] Understandably, since each node in the distributed database stores a tag table that corresponds one-to-one with each tag, tag data can be written to each tag table in parallel on each target node when it is written. When reading, it is also convenient to use parallel processing and load balancing. This embodiment of the invention overcomes the technical problem of concurrency conflicts that occur when the distributed database uses a built-in table engine to deal with ultra-large-scale tag data writing and reading, improves the efficiency of writing and reading, shortens the writing and reading time, and avoids writing and reading timeouts that affect business operations.

[0105] According to another aspect of the embodiments of the present invention, such as Figure 9 As shown, a data writing device 900 is provided, comprising:

[0106] The target determination module is used to determine the target node and target tag table corresponding to the data to be written based on the user identifier and tag identifier of the data to be written; wherein, the data to be written includes tag data under at least one tag, and the target node stores a tag table that corresponds one-to-one with each tag;

[0107] The data writing module is used to generate writing tasks corresponding to each of the tag data, and to assign each writing task to a thread, so that each thread writes the tag data to the target tag table respectively; wherein, each thread executes each writing task in parallel, and the writing tasks corresponding to tag data that are the same in both the target node and the target tag table are assigned to the same thread.

[0108] Figure 10 An exemplary system architecture 1000 is shown that can be applied to the data writing, data reading methods or data writing apparatus of embodiments of the present invention.

[0109] like Figure 10As shown, system architecture 1000 may include terminal devices 1001, 1002, and 1003, network 1004, and server 1005. Network 1004 is used as a medium to provide communication links between terminal devices 1001, 1002, and 1003 and server 1005. Network 1004 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0110] Users can use terminal devices 1001, 1002, and 1003 to interact with server 1005 via network 1004 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 1001, 1002, and 1003, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0111] Terminal devices 1001, 1002, and 1003 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0112] Server 1005 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 1001, 1002, and 1003 (this is just an example). The backend management server can analyze and process received data such as credit data and user profiles, and then feed the processing results back to the terminal devices.

[0113] It should be noted that the data writing method and data reading method provided in the embodiments of the present invention are generally executed by the server 1005, and correspondingly, the data writing device is generally set in the server 1005.

[0114] It should be understood that Figure 10 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0115] The following is for reference. Figure 11 It shows a schematic diagram of the structure of a computer system 1100 suitable for implementing a terminal device of the present invention. Figure 11 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0116] like Figure 11As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1102 or programs loaded from storage section 1108 into random access memory (RAM) 1103. The RAM 1103 also stores various programs and data required for the operation of the system 1100. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1106 is also connected to the bus 1104.

[0117] The following components are connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 110Y including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1110 as needed so that computer programs read from them can be installed into storage section 1108 as needed.

[0118] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by central processing unit (CPU) 1101, it performs the functions defined above in the system of this invention.

[0119] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0121] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a target determination module and a data writing module. The names of these modules do not necessarily limit the module itself; for example, the data writing module can also be described as "a module that writes data according to a mapping relationship."

[0122] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0123] Based on the user identifier and tag identifier of the data to be written, the target node and target tag table corresponding to the data to be written are determined; wherein, the data to be written includes tag data under at least one tag, and the target node stores a tag table that corresponds one-to-one with each tag;

[0124] Write tasks corresponding to each of the aforementioned tag data are generated and assigned to each thread, so that each thread writes the tag data to the target tag table respectively; wherein, each thread executes each write task in parallel, and write tasks corresponding to tag data that are the same in both the target node and the target tag table are assigned to the same thread.

[0125] In another aspect, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0126] According to the technical solution of the embodiments of the present invention, the present invention has the following advantages or beneficial effects: because the nodes of the distributed database store tag tables corresponding one-to-one with each tag, tag data can be written to each tag table in parallel on each target node when writing, and parallel processing is also convenient when reading. Therefore, it overcomes the technical problem of concurrency conflicts that occur when using a built-in table engine in the distributed database in the prior art when facing ultra-large-scale tag data writing and reading. The embodiments of the present invention improve the efficiency of writing and reading, shorten the writing and reading time, and avoid the impact of writing and reading timeouts on business operations by writing tag data to each tag table in parallel on each target node.

[0127] The specific embodiments described herein do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data writing method, characterized in that, include: Based on the user identifier and tag identifier of the data to be written, the target node and target tag table corresponding to the data to be written are determined; wherein, the data to be written includes tag data under at least one tag, and the target node stores a tag table that corresponds one-to-one with each tag; Write tasks corresponding to each of the aforementioned tag data are generated separately, and each write task is assigned to a different thread so that each thread writes the tag data to the target tag table. The threads execute each write task in parallel, and write tasks corresponding to tag data that are identical in both the target node and the target tag table are assigned to the same thread.

2. The method according to claim 1, characterized in that, Based on the user identifier and tag identifier of the data to be written, determine the target node and target tag table corresponding to the data to be written, including: The target node corresponding to the data to be written is determined based on the user identifier of the data to be written. For each piece of tag data, the target tag table corresponding to the tag data is determined based on the tag identifier corresponding to the tag data.

3. The method according to claim 2, characterized in that, Based on the user identifier of the data to be written, determine the target node corresponding to the data to be written, including: Obtain the user serial number corresponding to the user identifier of the data to be written, wherein the user serial number is obtained by sorting each user identifier according to the first sorting rule; Based on the total number of nodes and the user sequence number, and by calculating the node sequence number according to the horizontal sharding algorithm, the node corresponding to the node sequence number is determined as the target node corresponding to the data to be written.

4. The method according to claim 1, characterized in that, Based on the user identifier and tag identifier of the data to be written, determine the target node and target tag table corresponding to the data to be written, including: Obtain the user serial number corresponding to the user identifier of the data to be written, wherein the user serial number is obtained by sorting each user identifier according to the first sorting rule; Obtain the tag number corresponding to the tag identifier of the data tag, wherein the tag number is obtained by sorting each tag identifier according to the second sorting rule; Based on the total number of nodes, the user serial number, and the tag serial number, the target tag table serial number is calculated, thereby determining the target node and target tag table corresponding to the tag data.

5. The method according to claim 4, characterized in that, Also includes: A tag matrix is ​​constructed using each tag and each node as rows and columns, respectively. Each element in the tag matrix represents a tag table index, and the tag table indexes are obtained according to the third sorting rule. Based on the total number of nodes, the user serial number, and the tag serial number, the target tag table serial number is calculated to determine the target node and target tag table corresponding to the tag data, including: Subtract one from the tag number and multiply by the total number of nodes to obtain the first encoding value. Subtract one from the user number and divide by the total number of nodes to obtain the second encoding value. Add the first encoding value, the second encoding value and one to obtain the target tag table number. The target node and target label table corresponding to the label data are determined based on the target label table sequence number.

6. A data reading method, characterized in that, For reading data written using the method described in any one of claims 1-5, including: The user identifier and tag identifier are parsed from the data read request, and the target tag table is determined based on the user identifier and the tag identifier; Read the target label data from the target label table.

7. A data writing device, characterized in that, include: The target determination module is used to determine the target node and target tag table corresponding to the data to be written based on the user identifier and tag identifier of the data to be written; wherein, the data to be written includes tag data under at least one tag, and the target node stores a tag table that corresponds one-to-one with each tag; The data writing module is used to generate writing tasks corresponding to each of the tag data, and to assign each of the writing tasks to each thread, so that each thread writes the tag data to the target tag table respectively; wherein, each thread executes each writing task in parallel, and the writing tasks corresponding to tag data that are the same in both the target node and the target tag table are assigned to the same thread.

8. An electronic processing device, characterized in that, include: One or more processors; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images