Data processing method and device

Abstract

The present invention discloses a data processing method and device, relating to the field of computer technology. A specific implementation of the method includes: receiving a data processing request from a business layer via a first component; the first component and the business layer are mutually independent; determining, based on the data processing request, a database to be operated corresponding to the data to be processed; determining a synchronization method for the data to be processed based on the operational attributes of the database to be operated; and synchronizing, via the first component, the database to be operated with data in a Redis cache based on the synchronization method. This implementation can separate data synchronization from the business layer, allowing the business layer to focus on business development, reducing development costs, improving data synchronization efficiency, and enhancing user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] MySQL is a relational database management system that stores data in separate tables, offering high data processing speed and flexibility. Redis (Remote Dictionary Server) is a high-performance key-value database that supports storage of multiple value types.

[0003] When synchronizing MySQL data to Redis, developers usually develop code at the business layer to write data to MySQL and Redis.

[0004] In the existing method of synchronizing MySQL data to Redis, the business layer logic is coupled with the data synchronization work, making the business layer logic complex and prone to errors. This in turn leads to data synchronization failures, affecting subsequent query operations and other operations, and poor user experience. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a data processing method and device that can extract data synchronization from the business layer, allowing the business layer to focus on business development, reducing development costs, improving data processing efficiency, and enhancing user experience.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0007] receiving, through a first component, a data processing request sent by a business layer, the data processing request indicating data to be processed and an operation instruction for the data to be processed; the first component is independent of the business layer;

[0008] Determining, according to the data processing request, a database to be operated corresponding to the data to be processed;

[0009] Determining a synchronization method for the data to be processed according to the operation attributes of the database to be operated;

[0010] The first component synchronizes the database to be operated with the data in the Redis cache according to the synchronization method.

[0011] Optionally, when the synchronization mode is a dual-write mode, the data processing request is sent to the database to be operated and the Redis cache through the first component, so that the database to be operated and the Redis cache simultaneously process the data to be processed according to the operation instruction.

[0012] Optionally, it also includes:

[0013] Send a metadata request to the Etcd cluster, and receive metadata returned by the Etcd cluster according to the metadata request;

[0014] The first component parses the metadata to obtain a data processing request corresponding to the Redis cache, and sends the request to the Redis cache so that the Redis cache processes the data to be processed according to the operation instruction indicated by the corresponding data processing request.

[0015] Optionally, when the synchronization mode is a non-double-write mode, the data processing request is sent to the database to be operated through the first component, so that the database to be operated synchronizes the data in the Redis cache according to the processing result of the data to be processed.

[0016] Optionally, synchronizing the data in the Redis cache according to the processing result includes:

[0017] Analyzing the processing result by the second component to determine the updated data in the database to be operated;

[0018] The data in the Redis cache is synchronized according to the updated data.

[0019] Optionally, analyzing the processing result by the second component to determine the updated data in the database to be operated includes:

[0020] Send a metadata request to the Etcd cluster, and receive metadata returned by the Etcd cluster according to the metadata request;

[0021] Parsing detection point information in the shard information from the metadata by the second component;

[0022] The updated data in the database to be operated is determined according to the detection point information.

[0023] Optionally, synchronizing the data in the Redis cache according to the updated data includes:

[0024] Parsing synchronization information synchronized to the Redis cache from the metadata through the second component;

[0025] Determining data to be synchronized corresponding to the synchronization information from the updated data;

[0026] Synchronize the data in the Redis cache according to the data to be synchronized.

[0027] Optionally, the second component corresponds one-to-one to multiple nodes of the database to be operated, and the second component is deployed in the corresponding node.

[0028] According to another aspect of an embodiment of the present invention, there is provided a data processing apparatus, including:

[0029] a receiving module, configured to receive, through a first component, a data processing request sent by a business layer, the data processing request indicating data to be processed and an operation instruction for the data to be processed; the first component is independent of the business layer;

[0030] A request analysis module, configured to determine, based on the data processing request, a database to be operated corresponding to the data to be processed;

[0031] A data processing module, configured to determine a synchronization method for the data to be processed according to an operation attribute of the database to be operated;

[0032] A synchronization module is used to synchronize the database to be operated with the data in the Redis cache through the first component according to the synchronization method.

[0033] According to another aspect of an embodiment of the present invention, there is provided a data processing electronic device, comprising:

[0034] one or more processors;

[0035] a storage device for storing one or more programs,

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided by the present invention.

[0037] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data processing method provided by the present invention is implemented.

[0038] An embodiment of the above invention has the following advantages or beneficial effects: because the synchronization mode is determined by the vtgate component, when the synchronization mode is the dual-write mode, the data to be processed is written into the MySQL database and the Redis cache at the same time through the vtgate component; when the synchronization mode is not the dual-write mode, the data to be processed is written into the MySQL database. After the writing is completed, the data filtered out according to the filtering rules is synchronized to the Redis cache through the Vttablet component. Therefore, the technical problem that the business layer logic is complex and prone to errors, which leads to data synchronization failure, affects subsequent queries and other operations, and has a poor user experience is overcome, thereby achieving the technical effect of being able to extract data synchronization from the business layer, allowing the business layer to focus on business development, reducing development costs, improving data processing efficiency, and enhancing user experience.

[0039] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0041] Figure 1 An exemplary system architecture diagram of a data processing method or a data processing device suitable for application in an embodiment of the present invention is shown;

[0042] Figure 2 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of the main flow of a data processing method according to another embodiment of the present invention;

[0044] Figure 4 is a schematic diagram of the main process of a data synchronization method according to yet another embodiment of the present invention;

[0045] Figure 5 is an exemplary system architecture diagram of a data processing system according to an embodiment of the present invention;

[0046] Figure 6 is a schematic diagram of the main process of a method for creating a data processing system according to an embodiment of the present invention;

[0047] Figure 7 is a schematic diagram of the main process of the data query method according to an embodiment of the present invention;

[0048] Figure 8 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0049] Figure 9It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0051] Figure 1 FIG. 1 shows an exemplary system architecture diagram of a data processing method or a data processing device suitable for use in an embodiment of the present invention. Figure 1 As shown, an exemplary system architecture of the data processing method or data processing device according to an embodiment of the present invention includes:

[0052] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0053] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as data processing applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0054] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0055] The server 105 may be a server that provides various services, such as a backend management server that supports data processing websites browsed by users using the terminal devices 101, 102, and 103. The backend management server may analyze and process received data, such as data processing requests, and feedback the processing results (e.g., successful data synchronization) to the terminal devices 101, 102, and 103.

[0056] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 105 , and accordingly, the data processing device is generally provided in the server 105 .

[0057] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0058] Figure 2 is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention, such as Figure 2 As shown, the data processing method of the present invention includes:

[0059] Step S201: receiving a data processing request sent by a business layer through a first component, wherein the data processing request indicates data to be processed and operation instructions for the data to be processed; the first component and the business layer are independent of each other.

[0060] In an embodiment of the present invention, the first component may be a vtgate component, which receives data processing requests from the business layer through the vtgate component. The vtgate component is independent of the business layer. The data processing request indicates the data to be processed, an operation instruction for the data to be processed, and a synchronization flag. For example, the operation instruction may be deletion, addition, modification, etc.

[0061] Step S202: determining the database to be operated corresponding to the data to be processed according to the data processing request.

[0062] In an embodiment of the present invention, the synchronization identifier of the data processing request indicates the database to be operated corresponding to the data to be processed. The database to be operated corresponding to the data to be processed can be determined based on the synchronization identifier. For example, the database to be operated can be a MySQL database.

[0063] Step S203: determining a synchronization method for the data to be processed according to the operation attributes of the database to be operated.

[0064] In an embodiment of the present invention, the operation attributes of the database to be operated include: writing the data to be processed into the MySQL database and the Redis cache at the same time; writing the data to be processed into the MySQL database, and after the writing is completed, synchronizing the data written into the MySQL database to the Redis cache.

[0065] In an embodiment of the present invention, determining the synchronization method of the data to be processed based on the operation attributes of the database to be operated includes: when the operation attribute of the database to be operated is to write the data to be processed into the MySQL database and the Redis cache at the same time, determining that the synchronization method of the data to be processed is a dual-write method; when the operation attribute of the database to be operated is to write the data to be processed into the MySQL database, and after the writing is completed, synchronize the data written to the MySQL database to the Redis cache, determining that the synchronization method of the data to be processed is a non-dual-write method.

[0066] Step S204: Synchronize the database to be operated with the data in the Redis cache through the first component according to the synchronization method.

[0067] In an embodiment of the present invention, when the synchronization mode is a dual-write mode, the data processing request is sent to the database to be operated and the Redis cache at the same time through the first component, so that the database to be operated and the Redis cache process the data to be processed at the same time according to the operation instruction.

[0068] like Figure 3 As shown, a data processing method according to another embodiment of the present invention includes the following steps:

[0069] Step S301: Send a metadata request to the Etcd cluster through the first component, and receive metadata returned by the Etcd cluster according to the metadata request.

[0070] In an embodiment of the present invention, the metadata request may include a Redis cache synchronization rule request, etc., and the metadata returned by the received Etcd cluster according to the metadata request may include: Redis cache synchronization rules, etc.

[0071] Step S302: The first component parses the metadata to obtain synchronization rules for synchronization to the Redis cache.

[0072] In an embodiment of the present invention, synchronization rules for synchronizing to a Redis cache may include basic information such as the Redis cache's IP address, username, and password, and the Redis cache to be written to may be determined based on the basic information. They may also include filtering rules for the Redis cache and the data to be synchronized may be determined based on the filtering rules.

[0073] For example, the vtgate component is used to read the value of the key / keyspace / cache / test_keyspace / redis_cluster / filters in the Etcd cluster to obtain all the filtering rules in the test_keyspace rule library.

[0074] Step S303: According to the synchronization rule, the first component determines whether the data to be processed is data to be synchronized. If yes, go to step S304; if not, go to step S307.

[0075] In this embodiment of the present invention, the vtgate component determines whether the data to be processed is data to be synchronized based on the basic information in the synchronization rules and the filtering rules corresponding to the basic information. If so, the process proceeds to step S304; if not, the process proceeds to step S307. If the data to be processed meets the filtering rules, the data to be processed is determined to be data to be synchronized; if the data to be processed does not meet the filtering rules, the data to be processed is determined not to be data to be synchronized.

[0076] Exemplarily, the filtering rule may be: data with type='vip' in the customer table and type='*' in the dict table, and the data to be synchronized is data with type='vip' in the customer table and type='*' in the dict table.

[0077] Step S304: the database to be processed and the Redis cache process the data to be processed.

[0078] In an embodiment of the present invention, after each node of the database to be operated receives the data processing request sent by the vtgate component and the Redis cache receives the data processing request sent by the vtgate component, the data to be processed is written according to the operation instructions for the data to be processed indicated by the data processing request.

[0079] In an embodiment of the present invention, when the database to be operated is a MySQL database, after each node of the MySQL database receives the data processing request sent by the vtgate component and the Redis cache receives the data processing request sent by the vtgate component, the data to be processed is written according to the operation instructions for the data to be processed indicated by the data processing request.

[0080] Step S305, determine whether the simultaneous writing is successful. If the Redis cache writing fails, go to step S306.

[0081] In this embodiment of the present invention, it is determined whether both the database to be processed and the Redis cache are successfully written simultaneously. If the database to be operated on is a MySQL database, it is determined whether both the MySQL database and the Redis cache are successfully written simultaneously. If the MySQL database is successfully written but the Redis cache is not, the process proceeds to step S306. Otherwise, an error message indicating the processing failure is sent to the business layer so that the business layer can resend the data processing request.

[0082] Step S306: Put the data processing request that failed to be written into the retry queue of RabbitMQ for retry.

[0083] In an embodiment of the present invention, when the Vtgate component implements dual writing to the MySQL database and Redis cache, priority is given to ensuring that both are written successfully. If the MySQL database write succeeds but the Redis cache write fails, the failed write data will be placed in the MQ delay queue, and the Vtgate component will retry writing to the Redis cache again.

[0084] Step S307: the database to be processed processes the data to be processed.

[0085] In the embodiment of the present invention, after receiving the data processing request sent by the vtgate component, each node of the database to be operated writes the data to be processed according to the operation instructions for the data to be processed indicated in the data processing request.

[0086] In an embodiment of the present invention, when the database to be operated is a MySQL database, after each node of the MySQL database receives a data processing request sent by the vtgate component, it writes the data to be processed according to the operation instructions for the data to be processed indicated by the data processing request.

[0087] The existing data synchronization process requires business personnel to develop dual-write code for MySQL and Redis, as well as logic for determining read and write data sources, at the business layer. For example, the business layer needs to configure MySQL and Redis, and determine the source of the read and write data when reading and writing data. However, this business layer code development increases the complexity of the business layer logic, making it prone to errors and even crashes.

[0088] In an embodiment of the present invention, the data synchronization method of the second embodiment of the present invention is used to separate data processing work from the business layer logic, thereby avoiding the complex logic of the business layer, ensuring the stability of the business layer, and preventing the business layer from crashing.

[0089] In an embodiment of the present invention, when the synchronization mode is non-dual write, the first component sends a data processing request to the database to be operated, causing the database to synchronize the data in the Redis cache based on the processing results of the data to be processed. For example, when the first component is the vtgate component and the database to be operated is a MySQL database, the vtgate component sends the data processing request to the MySQL database, causing the MySQL database to synchronize the data in the Redis cache based on the processing results of the data to be processed.

[0090] like Figure 4 As shown, a data synchronization method according to another embodiment of the present invention includes the following steps:

[0091] Step S401: Send a first metadata request to the Etcd cluster through a first component, and receive first metadata returned by the Etcd cluster according to the first metadata request.

[0092] In an embodiment of the present invention, the first metadata request may include a request for the number of shards of the database to be operated, a request for the data interval range stored in each shard, a request for the node IP of each shard, etc. The first metadata returned by the received Etcd cluster based on the first metadata request may include: the number of shards of the database to be operated, the data interval range stored in each shard, the node IP of each shard, etc.

[0093] In an embodiment of the present invention, when the database to be operated is a MySQL database, the first metadata request may include a request for the number of shards of the MySQL database, a request for the data interval range stored in each shard, an IP request for the MySQL node of each shard, etc. The first metadata returned by the received Etcd cluster based on the first metadata request may include: the number of shards of the MySQL database, the data interval range stored in each shard, the IP of the MySQL node of each shard, etc.

[0094] Step S402: Send the data processing request to each shard of the database to be operated through the first component.

[0095] In an embodiment of the present invention, the first component may send a data processing request to a corresponding shard of the database to be operated based on the node IP of the shard.

[0096] In an embodiment of the present invention, when the database to be operated is a MySQL database, the first component can send a data processing request to the corresponding shard of the MySQL database based on the IP of the MySQL node of the shard.

[0097] In an embodiment of the present invention, the database to be operated may include multiple shards, each of which includes multiple nodes. If the database to be operated is a MySQL database, the MySQL database may include multiple shards, each of which includes multiple MySQL nodes, such as a MySQL master node and a MySQL slave node. MySQL master-slave replication is used for disaster recovery and read-write separation. If the master node fails, a slave node can be switched to the master node to continue providing services.

[0098] In an embodiment of the present invention, the database to be operated may include multiple shard tables and non-shard tables, and the data processing request corresponding to the non-shard table is sent to the first shard, and the data processing request of the shard table is sent to other shards.

[0099] In an embodiment of the present invention, when the database to be operated is a MySQL database, the MySQL database may include a sharded table customer table and a non-sharded table dict table. The data processing request for the dict table is sent to the first shard through the vtgate component; according to the sharding rules, the data processing request for the customer table is sent to other calculated shards. For example, the data processing request may be a write operation to the dict table, and the SQL written is: insert into dict(name,value)values("menu","system"), which is sent to the first shard. For another example, the data processing request may be a write operation to the customer table, and the SQL written is: insert into customer(name,email,type)values("lily","lily@jd.com","vip"), which is sent to other shards.

[0100] Step S403: After receiving the data processing request, the operation database processes the data to be processed according to the data processing request.

[0101] In an embodiment of the present invention, the second component may be a Vttablet component. When the database to be operated is a MySQL database, after receiving the data processing request sent by the vtgate component, each node of the MySQL database writes the data to be processed according to the operation instructions for the data to be processed indicated in the data processing request.

[0102] Step S404: Obtain data processing results through the second component.

[0103] In the embodiment of the present invention, the data processing result may be a log file obtained when the MySQL database processes the data to be processed according to the operation instruction, such as a binary log file MySQL Binlog.

[0104] Step S405: Send a second metadata request to the Etcd cluster through the second component, and receive the second metadata returned by the Etcd cluster according to the metadata request.

[0105] In an embodiment of the present invention, the second metadata request may include a detection point information request for each shard, a synchronization information request for the Redis cache, etc. The second metadata returned by the received Etcd cluster based on the second metadata request may include detection point information for each shard, synchronization information for the Redis cache, etc.

[0106] In this embodiment of the present invention, the second component corresponds one-to-one with multiple nodes of the database to be operated, and the second component is deployed within the corresponding node. If the database to be operated is a MySQL database, each MySQL node includes a Vttablet component, which corresponds one-to-one with the MySQL master node and MySQL slave node. This allows the data processing results of the MySQL master and MySQL slave nodes to be analyzed and the updated data to be determined.

[0107] In this embodiment of the present invention, the Etcd cluster is essentially a highly available distributed key-value database that uses the Raft protocol as its internal consensus algorithm and can be implemented in the Go language. Metadata can also include information such as the data range stored in each shard.

[0108] For example, the Vttablet component reads the value of the key / keyspace / cache / test_keyspace / redis_cluster in the Etcd cluster to obtain the number of MySQL database shards, the MySQL master node information for each shard, the MySQL slave node information for each shard, and the checkpoint information for each shard. A checkpoint can be represented by a checkpoint.

[0109] Step S406: parse the detection point information in the fragment information from the metadata through the second component.

[0110] In an embodiment of the present invention, the Vttablet component parses the detection point information in the shard information from the metadata. The detection point information may include: the number of detection points on each shard, the files and locations detected by the detection points, etc.

[0111] In the embodiment of the present invention, for example, the MySQL binlog records the MySQL operation records, and MySQL executes the following three SQL statements:

[0112] 1. insert into test(id,value)values(1,"system");

[0113] 2. insert into test(id,value)values(2,"system");

[0114] 3. insert into test(id,value)values(3,"system");

[0115] Operations 1 and 2 are recorded in the binlog01 file, and operation 3 is recorded in the binlog02 file. In the checkpoint["binlog":xxx,"pos":xxx] file, "binlog" corresponds to the binlog01 and binlog02 files, and "pos" corresponds to the position of the first and second operation records.

[0116] Step S407 , the second component is used to judge the detection point information to confirm whether it is the first detection. If so, go to step 408 ; if not, go to step 409 .

[0117] In this embodiment of the present invention, the Vttablet component determines whether this is the first detection based on the detection point information. If so, it is determined to be an initialization update and the process proceeds to step 408. If not, it is determined to be an incremental update and the process proceeds to step 409. If the value of the detection point is null, it is determined to be the first detection; if the value of the detection point is not null, it is determined to be a non-first detection.

[0118] For example, the checkpoint is ["binlog":xxx,"pos":xxx]. If the binlog value is empty and the pos value is 0, that is, ["binlog":,"pos":0], it is determined to be the first detection. If the binlog value is not empty and the pos value is not 0, it is determined not to be the first detection.

[0119] Step S408 , initializing the update, determining the updated data in the processing result through the second component, and then going to step 410 .

[0120] In an embodiment of the present invention, when initializing the update, the second component initializes the value of the detection point and assigns the current value of the detection point to the detection point. The second component determines the data in the processing result corresponding to the value of the detection point based on the value of the detection point, and further determines the data to be updated.

[0121] In an embodiment of the present invention, when the data processing result is a log file of a MySQL database, the MySQL Binlog is parsed in real time by the second component to determine the updated data in the database to be operated.

[0122] For example, when initializing an update, the Vttablet component initializes the value of checkpoint["binlog":,"pos":0], assigns the file name "binlog-100" and file position "xx-100" currently being parsed by the checkpoint to checkpoint["binlog":,"pos":0], sends an assignment request to the Etcd cluster, and assigns the value of the key value / keyspace / cache / test_keyspace / redis_cluster / checkpoint / shard1 to ["binlog":binlog-100,"pos":xx-100]. Based on the value of the checkpoint, the Vttablet component determines the data of xx-100 in MySQL Binlog-100 corresponding to the checkpoint value, and then determines the updated data.

[0123] Step S409 , incremental update, determines the updated data in the processing result through the second component, and goes to step 410 .

[0124] In an embodiment of the present invention, during incremental update, the second component determines the processing result corresponding to the value of the detection point based on the value of the detection point, detects the corresponding processing result, determines whether the detected data has been updated, and determines the updated data.

[0125] In an embodiment of the present invention, when the data processing result is a log file of a MySQL database, the MySQL Binlog is parsed in real time by the second component to determine the updated data in the database to be operated.

[0126] For example, during incremental updates, the Vttablet component determines the binlog file and file location corresponding to the checkpoint value based on the value of checkpoint["binlog":binlog-100,"pos":xx-100], detects the xx-100 location of the binlog-100 file, determines whether the data at the xx-100 location of the binlog-100 file has been updated, and determines the updated data.

[0127] Step S410: parse the second metadata through the second component to obtain synchronization information to be synchronized to the Redis cache.

[0128] In an embodiment of the present invention, the synchronization information synchronized to the Redis cache may include basic information such as the Redis cache's IP address, username, and password, and the Redis cache to be synchronized is determined based on the basic information. It may also include filtering rules for the Redis cache, and the data to be synchronized to the Redis cache from the updated data is determined based on the filtering rules.

[0129] Exemplarily, the synchronization information synchronized to the Redis cache is parsed from the metadata by the Vttablet component.

[0130] For example, the Vttablet component reads the value of the key / keyspace / cache / test_keyspace / redis_cluster in the Etcd cluster to obtain the basic information of the Redis cache, such as the IP address, user name, and password. The Vttablet component reads the value of the key / keyspace / cache / test_keyspace / redis_cluster / filters in the Etcd cluster to obtain all the filter rules in the test_keyspace rule library.

[0131] In the embodiment of the present invention, step S410 may be performed before step S405, or step S410 and step S405 may be performed simultaneously.

[0132] Step S411: According to the synchronization information, the second component determines the data to be synchronized corresponding to the synchronization information from the updated data.

[0133] In an embodiment of the present invention, the Vttablet component filters the data to be synchronized from the updated data according to the basic information in the synchronization information and the filtering rules corresponding to the basic information.

[0134] Exemplarily, the filtering rules are: data with type = 'vip' in the customer table and all data in the dict table; the data to be synchronized is the data with type = 'vip' in the customer table and all data in the dict table in the updated data. Furthermore, when filtering the data to be synchronized, the Vttablet component determines whether the customer table and dict table exist in the MySQL database. If so, it filters the data with type = 'vip' in the customer table and determines that the data with type = 'vip' in the customer table and all data in the dict table are the data to be synchronized. If not, it returns an error prompt so that the MySQL database can create the customer table and dict table according to the error prompt. The key values ​​corresponding to the filtering rules can be / keyspace / cache / test_keyspace / redis_cluster / filters / customer_filter and / keyspace / cache / test_keyspace / redis_cluster / filters / dict_filter.

[0135] For example, the script for creating the customer table in the MySQL database and processing the data to be synchronized may be:

[0136] create table customer(customer_id bigint not null auto_increment,namevarchar(128),email varchar(128),type varchar(64),primary key(customer_id));

[0137] ENGINE=InnoDB;

[0138] insert into customer(name,email,type)values("lily","lily@jd.com","vip");

[0139] / / If this data satisfies type='vip', then synchronize the data to the Redis cluster.

[0140] insert into customer(name,email)values("tom","tom@jd.com");

[0141] / The type of this data is empty and does not satisfy type='vip', so the change event of this data is discarded / .

[0142] For example, the script for creating a dict table in the MySQL database and processing the data to be processed may be:

[0143] create table dict(dict_id bigint not null auto_increment,name varchar(128),value varchar(128),primary key(customer_id));

[0144] ENGINE=InnoDB;

[0145] insert into dict(name,value)values("menu","system").

[0146] Step S412: Synchronize the data in the Redis cache according to the data to be synchronized.

[0147] In an embodiment of the present invention, the Vttablet component sends the filtered data to be synchronized to the Redis cache, so that the Redis cache writes the data to be synchronized, thereby realizing synchronization of the data to be synchronized.

[0148] In the existing data synchronization process, it is necessary to call open source services such as canal, build a service to parse MySQL binlog in canal, and then synchronize data to the Redis cluster. However, using open source services such as canal increases the development cost of business R&D personnel and requires real-time maintenance, which is costly. Among them, Canal is an open source project that parses incremental logs based on the MySQL database and provides incremental data subscription and consumption functions.

[0149] In an embodiment of the present invention, through the data synchronization method of the second embodiment of the present invention, the Vttablet component can parse the MySQL binlog and synchronize the data to the Redis cluster without calling other services. The Canal component in the original architecture can be removed, reducing the development and maintenance costs of business personnel.

[0150] like Figure 5 As shown, an embodiment of the present invention discloses a data processing system that can be applied to a vitness cluster, including:

[0151] The data processing request sent by the business layer is received through the vtgate component 5011. Based on the data processing request, the MySQL database 501 corresponding to the data to be processed is determined. The vtgate component 5011 determines the synchronization method of the data to be processed based on the operation attributes of the MySQL database 501. Specifically, when the operation attribute of the MySQL database 501 is to write the data to be processed into the MySQL database and the Redis cache at the same time, the synchronization method of the data to be processed is determined to be a double write method; when the operation attribute of the MySQL database 501 is to write the data to be processed into the MySQL database 501 and, after the writing is completed, synchronize the data written into the MySQL database 501 to the Redis cache 502, the synchronization method of the data to be processed is determined to be a non-double write method.

[0152] When the synchronization mode is dual-write, vtgate component 5011 sends a metadata request to Etcd cluster 503 and receives metadata returned by Etcd cluster 503 based on the metadata request. vtgate component 5011 parses the metadata to determine synchronization rules for Redis cache 502. Based on the synchronization rules, vtgate component 5011 determines whether the pending data is data to be synchronized. If so, MySQL database 501 and Redis cache 502 process the pending data. If not, MySQL database 501 processes the pending data.

[0153] After processing the data to be processed, the MySQL database 501 and the Redis cache 502 determine whether the writing is successful at the same time. If the writing to the Redis cache fails, the data processing request that failed to be written is placed in the RabbitMQ queue 5021 for retry.

[0154] When the synchronization mode is dual write mode, a first metadata request is sent to the Etcd cluster 503 through the vtgate component 5011, and metadata returned by the Etcd cluster 503 according to the first metadata request is received. The vtgate component 5011 sends a data processing request to each shard 5012 of the MySQL database 501 based on the node IP of the shard 5012.

[0155] After MySQL database 501 receives a data processing request, the MySQL nodes in each shard 5012 process the data to be processed according to the data processing request. The data processing results are obtained through Vttablet component 5015. Vttablet component 5015 corresponds one-to-one with the MySQL nodes in each shard 5012 of MySQL database 501, and Vttablet component 5015 is deployed in the corresponding MySQL node.

[0156] A second metadata request is sent to the Etcd cluster 503 through the Vttablet component 5015, and the second metadata returned by the Etcd cluster 503 according to the metadata request is received. The detection point information in the shard information is parsed from the metadata through the Vttablet component 5015. The detection point information is judged by the Vttablet component 5015 to confirm whether it is the first detection. If so, the update is initialized, and the updated data in the processing result is determined by the Vttablet component 5015; if not, the incremental update is performed, and the updated data in the processing result is determined by the Vttablet component 5015. The synchronization information synchronized to the Redis cache is parsed from the metadata through the Vttablet component 5015. According to the synchronization information, the data to be synchronized corresponding to the synchronization information is determined from the updated data through the Vttablet component 5015, and sent to the Redis cache 502, so that the Redis cache 502 writes the data to be synchronized.

[0157] In an embodiment of the present invention, Vitess is a distributed MySQL toolset that can automatically shard and store MySQL data tables, rewrite single SQL queries into distributed ones and send them to multiple MySQL Servers. It is a database solution for deploying, expanding, and managing large MySQL instance clusters, and supports row caching (which is more efficient than MySQL's own caching) and replication fault tolerance.

[0158] In an embodiment of the present invention, through the data processing system of an embodiment of the present invention, dual writing of MySQL database and Redis cache can be realized through the Vtgate component, and parsing and real-time conditional filtering synchronization of MySQL binlog can be realized through the Vttablet component. The data synchronization operation from MySQL database to Redis cache originally controlled by business personnel is encapsulated into Vitess, realizing the integration of MySQL database and Redis cache data.

[0159] like Figure 6 As shown, the method for creating a data processing system according to an embodiment of the present invention includes the following steps:

[0160] Step S601: Create one or more MySQL databases.

[0161] Step S602: Write the created MySQL database information into the Etcd cluster.

[0162] In the embodiment of the present invention, the MySQL database information includes: MySQL database name, user name, password, filter conditions, sharding information, detection point information, etc. For example, the MySQL database name is test_keyspace, the user name is test_user, and the password is test_password.

[0163] In an embodiment of the present invention, the corresponding script for writing MySQL database information into the Etcd cluster may be:

[0164] {key: / keyspace / db / test_keyspace,

[0165] value:{shards:” / keyspace / db / test_keyspace / shard1”,”keyspace / db / test_keyspace / shard2”,

[0166] mysql_dbname:test_keyspace}}

[0167] {key: / keyspace / db / test_keyspace / shard1,

[0168] value:{hosts:”192.168.101.7”,”192.168.101.8”,”192.168.101.9”}}

[0169] {key: / keyspace / db / test_keyspace / shard2,

[0170] value:{hosts:”192.168.101.10”,”192.168.101.11”,”192.168.101.12”}}

[0171] Step S603: Create one or more Redis caches.

[0172] Step S604: Write the created Redis cache information into the Etcd cluster.

[0173] In the embodiment of the present invention, the Redis cache information includes: Redis cache IP, user name, password, etc. For example, the Redis cache user name is redis_user and the password is redis_password.

[0174] Step 605, determine whether the synchronization mode is the dual-write mode, if yes, go to step S606; if not, go to step S607.

[0175] Step S606: Send business layer MySQL database information.

[0176] In the embodiment of the present invention, the MySQL database information sent to the business layer includes: the name of the MySQL database, the user name, the password, the connection string, etc. For example, the MySQL database name is test_keyspace, the user name is test_user, the password is test_password, and the connection string is test_keyspace.xiaoming.com.

[0177] In the embodiment of the present invention, after the system is created, the first writing into the MySQL database corresponds to the initialization update of the data processing method in the second embodiment of the present invention.

[0178] In the embodiment of the present invention, the non-dual write method uses Vttablet to parse MySQL Binlog in real time and write it into Redis cache, without the need for RabbitMQ queue, and the system creation is completed.

[0179] Step S607: Create a RabbitMQ queue, write the RabbitMQ queue information into the Etcd cluster, and go to step S606.

[0180] In an embodiment of the present invention, through the method for creating a data processing system in an embodiment of the present invention, data processing work can be separated from business layer logic, avoiding the complexity of business layer logic, simplifying the load of business R&D code, ensuring the stability of the business layer, and preventing business layer crashes; determining different data synchronization methods based on different database attributes can further improve data processing efficiency, ensure the stability of subsequent operations, and enhance user experience.

[0181] like Figure 7 As shown, the data query method according to an embodiment of the present invention includes the following steps:

[0182] Step S701: Receive an SQL query statement by receiving a first component.

[0183] In the embodiment of the present invention, the SQL query statement is received by the receiving module of the vtgate component, and the SQL statement is parsed by the SQL parsing module of the vtgate component.

[0184] For example, the SQL statement is a write operation for the customer table under the test_keyspace library.

[0185] Step S702: Based on the SQL query statement, determine whether the query target is the database to be operated or the Redis cache. If it is the database to be operated, go to step S703; if it is the Redis cache, go to step S704.

[0186] In an embodiment of the present invention, the database to be operated on may be a MySQL database. Whether the query target is the MySQL database or the Redis cache is determined by whether the SQL query statement contains a hint annotation. The hint annotation may be / *cache* / . For example, a SQL statement to query the MySQL database may be: "select email from customer where name = "lily""; a SQL statement to query the Redis cache may be: "select / *cached* / emailfromcustomer where name = "lily"".

[0187] Step S703: Send a third metadata request to the Etcd cluster, and receive the third metadata returned by the Etcd cluster according to the metadata request. Connect to the database to be operated based on the third metadata, obtain the query result, and go to step S705.

[0188] In an embodiment of the present invention, the third metadata may include connection string information of a MySQL database. For example, the value of the key value / keyspace / db / test_keyspace in the Etcd cluster is read to obtain MySQL connection information and connect to MySQL.

[0189] For example, the result of name="lily" is read from the customer table of MySQL.

[0190] Step S704: Send a fourth metadata request to the Etcd cluster, and receive fourth metadata returned by the Etcd cluster according to the metadata request. Connect to the Redis cache based on the fourth metadata, obtain the query result, and then go to step S705.

[0191] In an embodiment of the present invention, the fourth metadata may include connection string information of the Redis cache. For example, the value of the key value / keyspace / cache / test_keyspace / redis_cluster in the Etcd cluster is read to obtain the connection information of the Redis cache and connect to Redis.

[0192] Exemplarily, all values ​​of key="lily" are read from Redis.

[0193] Step S705: Return the query result through the first component.

[0194] In an embodiment of the present invention, the query result is returned through the vtgate component.

[0195] In an embodiment of the present invention, through the data query method of an embodiment of the present invention, the required data can be queried directly through the first component without the need for logical judgment of the business layer, thereby avoiding complex business layer logic, simplifying the business R&D code load, ensuring the stability of the business layer, preventing the business layer from crashing, further improving data query efficiency, and enhancing user experience.

[0196] Figure 8 is a schematic diagram of the main modules of the data processing device according to an embodiment of the present invention, such as Figure 8 As shown, the data processing device 800 of the present invention includes:

[0197] The receiving module 801 is used to receive a data processing request sent by the business layer through a first component, where the data processing request indicates the data to be processed and an operation instruction for the data to be processed; the first component is independent of the business layer.

[0198] In an embodiment of the present invention, the first component may be a vtgate component. The receiving module 801 receives data processing requests sent by the business layer via the vtgate component. The vtgate component is independent of the business layer. The data processing request indicates the data to be processed, an operation instruction for the data to be processed, and a synchronization flag. For example, the operation instruction may be deletion, addition, modification, etc.

[0199] The request analysis module 802 is configured to determine the database to be operated corresponding to the data to be processed according to the data processing request.

[0200] In an embodiment of the present invention, the synchronization identifier of the data processing request indicates the database to be operated corresponding to the data to be processed. The request analysis module 802 can determine the database to be operated corresponding to the data to be processed based on the synchronization identifier. For example, the database to be operated can be a MySQL database.

[0201] The data processing module 803 is configured to determine a synchronization method for the data to be processed according to the operation attributes of the database to be operated.

[0202] In an embodiment of the present invention, the operation attributes of the database to be operated include: writing the data to be processed into the MySQL database and the Redis cache at the same time; writing the data to be processed into the MySQL database, and after the writing is completed, synchronizing the data written into the MySQL database to the Redis cache.

[0203] In an embodiment of the present invention, the data processing module 803 determines the synchronization method of the data to be processed based on the operation attributes of the database to be operated, including: when the operation attribute of the database to be operated is to write the data to be processed into the MySQL database and the Redis cache at the same time, determining that the synchronization method of the data to be processed is a dual-write method; when the operation attribute of the database to be operated is to write the data to be processed into the MySQL database, and after the writing is completed, synchronize the data written to the MySQL database to the Redis cache, determining that the synchronization method of the data to be processed is a non-dual-write method.

[0204] The synchronization module 804 is used to synchronize the database to be operated with the data in the Redis cache through the first component according to the synchronization method.

[0205] In an embodiment of the present invention, when the synchronization mode is a dual-write mode, the synchronization module 804 sends the data processing request to the database to be operated and the Redis cache simultaneously through the first component, so that the database to be operated and the Redis cache can process the data to be processed simultaneously according to the operation instructions.

[0206] In an embodiment of the present invention, when the synchronization mode is non-dual write, the synchronization module 804 sends a data processing request to the database to be operated via the first component, causing the database to be operated to synchronize the data in the Redis cache based on the processing results of the data to be processed. For example, when the first component is the vtgate component and the database to be operated is a MySQL database, the data processing request is sent to the MySQL database via the vtgate component, causing the MySQL database to synchronize the data in the Redis cache based on the processing results of the data to be processed.

[0207] In an embodiment of the present invention, through modules such as the receiving module, the request analysis module, the data processing module and the synchronization module, the data processing work can be separated from the business layer logic, thereby avoiding the complexity of the business layer logic, simplifying the business development code load, ensuring the stability of the business layer, and preventing the business layer from crashing; according to different database properties, different data synchronization methods are determined, which can further improve data processing efficiency, ensure the stability of subsequent operations, and enhance user experience.

[0208] Figure 9 FIG. 1 is a schematic diagram of a computer system of a terminal device or server suitable for implementing an embodiment of the present invention. Figure 9 As shown, the computer system 900 of the terminal device or server according to the embodiment of the present invention includes:

[0209] A central processing unit (CPU) 901 can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the system 900 are also stored in the RAM 903. The CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0210] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, and the like; an output section 907 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card or a modem. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 910 as needed, so that computer programs read therefrom can be installed into the storage section 908 as needed.

[0211] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from a removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are performed.

[0212] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0213] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0214] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor including a receiving module, a request analysis module, a data processing module, and a synchronization module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the request analysis module may also be described as a module that analyzes the data processing request based on the received data processing request and determines the database to be operated corresponding to the data to be processed.

[0215] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: receiving a data processing request sent by the business layer through a first component, the data processing request indicating the data to be processed and the operation instructions for the data to be processed; the first component is independent of the business layer; determining the database to be operated corresponding to the data to be processed according to the data processing request; determining the synchronization method of the data to be processed according to the operation attributes of the database to be operated; synchronizing the database to be operated with the data in the Redis cache through the first component according to the synchronization method.

[0216] According to the technical solution of the embodiment of the present invention, data synchronization can be extracted from the business layer, allowing the business layer to focus on business development, reducing development costs, improving data synchronization efficiency, and enhancing user experience.

[0217] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: receiving, through a first component, a data processing request sent by a business layer, the data processing request indicating data to be processed and an operation instruction for the data to be processed, the operation instruction being a deletion instruction, an addition instruction, or a modification instruction; the first component being independent of the business layer; Determining, according to the data processing request, a database to be operated corresponding to the data to be processed; Determining a synchronization method for the data to be processed according to the operation attributes of the database to be operated; The first component synchronizes the database to be operated with the data in the Redis cache according to the synchronization method.

2. The method according to claim 1, characterized in that When the synchronization mode is a dual-write mode, the data processing request is sent to the database to be operated and the Redis cache through the first component, so that the database to be operated and the Redis cache simultaneously process the data to be processed according to the operation instruction.

3. The method according to claim 2, characterized in that Also includes: Send a metadata request to the Etcd cluster, and receive metadata returned by the Etcd cluster according to the metadata request; The first component parses the metadata to obtain a data processing request corresponding to the Redis cache, and sends the request to the Redis cache so that the Redis cache processes the data to be processed according to the operation instruction indicated by the corresponding data processing request.

4. The method according to claim 1, wherein When the synchronization mode is a non-double-write mode, the data processing request is sent to the database to be operated through the first component, so that the database to be operated synchronizes the data in the Redis cache according to the processing result of the data to be processed.

5. The method according to claim 4, characterized in that Synchronizing the data in the Redis cache according to the processing result includes: Analyzing the processing result by a second component to determine the updated data in the database to be operated; The data in the Redis cache is synchronized according to the updated data.

6. The method according to claim 5, characterized in that The analyzing the processing result by the second component to determine the updated data in the database to be operated includes: Send a metadata request to the Etcd cluster, and receive metadata returned by the Etcd cluster according to the metadata request; Parsing detection point information in the shard information from the metadata by the second component; The updated data in the database to be operated is determined according to the detection point information.

7. The method according to claim 6, characterized in that The step of synchronizing the data in the Redis cache according to the updated data includes: Parsing synchronization information synchronized to the Redis cache from the metadata through the second component; Determining data to be synchronized corresponding to the synchronization information from the updated data; Synchronize the data in the Redis cache according to the data to be synchronized.

8. The method according to any one of claims 5 to 7, characterized in that The second component corresponds one-to-one to multiple nodes of the database to be operated, and the second component is deployed in the corresponding node.

9. A data processing device, characterized in that: include: a receiving module, configured to receive, through a first component, a data processing request sent by a business layer, the data processing request indicating data to be processed and an operation instruction for the data to be processed, the operation instruction being a deletion instruction, an addition instruction, or a modification instruction; the first component being independent of the business layer; A request analysis module, configured to determine, based on the data processing request, a database to be operated corresponding to the data to be processed; A data processing module, configured to determine a synchronization method for the data to be processed according to an operation attribute of the database to be operated; A synchronization module is used to synchronize the database to be operated with the data in the Redis cache through the first component according to the synchronization method.

10. A data processing electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images