A data processing method, apparatus and device, and a storage medium

Abstract

The application discloses a data processing method, device and equipment and a storage medium. The method comprises the following steps: in the process of monitoring each shard of a distributed database, whether data skew exists is preliminarily determined according to the number of records of a candidate table on each shard; if yes, a target table meeting a preset monitoring condition is determined from the candidate table, and a target operation instruction for the target table is sent to a candidate shard storing the target table; the response time of each candidate shard is used to determine monitoring shards, and in the case that it is finally determined that data skew exists, data redistribution is performed according to the monitoring result of the slow SQL phenomenon of each monitoring shard. The technical scheme of the application can comprehensively and accurately monitor the data of each shard of the distributed database, accurately detect the data skew condition, and timely perform data redistribution, thereby providing strong support for database operation and maintenance.

Description

Technical Field

[0001] This invention relates to the field of distributed database data partitioning, and more particularly to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] A distributed database is a database system where data is physically distributed but logically centrally managed. Distributed databases offer transparency, data redundancy, and scalability. However, uneven distribution of key-value pairs in a distributed database can lead to uneven distribution of query actions, causing some nodes to become overloaded and degrading the system's performance. Therefore, automatic online data redistribution is crucial for distributed databases.

[0003] Therefore, how to comprehensively and accurately monitor the data in each shard of a distributed database, precisely detect data skew, and promptly redistribute data to provide strong support for database operation and maintenance is an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, device, and storage medium that can comprehensively and accurately monitor the data in each shard of a distributed database, precisely detect data skew, and promptly redistribute data, providing strong support for database operation and maintenance.

[0005] According to one aspect of the present invention, a data processing method is provided, comprising:

[0006] During the monitoring of each shard of the distributed database, the number of records of the candidate table on each shard is used to preliminarily determine whether there is data skew.

[0007] If so, the target table that meets the preset monitoring conditions is determined from the candidate table, and the target operation instruction for the target table is sent to the candidate shard that stores the target table.

[0008] Based on the response time of each candidate shard, the monitoring shard is determined, and based on the monitoring results of slow SQL phenomena in each monitoring shard, if data skew is finally determined, data redistribution is performed.

[0009] According to another aspect of the present invention, a data processing apparatus is provided, comprising:

[0010] The determination module is used to initially determine whether there is data skew based on the number of records of candidate tables on each shard during the monitoring of each shard of a distributed database.

[0011] The sending module is used to determine the target table that meets the preset monitoring conditions from the candidate table if the condition is met, and send the target operation instruction to the candidate fragment that stores the target table.

[0012] The redistribution module is used to determine the monitoring shards based on the response time of each candidate shard, and to redistribute the data based on the monitoring results of slow SQL phenomena in each monitoring shard, if data skew is finally determined to exist.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data processing method described in any embodiment of the present invention.

[0018] The technical solution of this invention, during the monitoring of each shard of a distributed database, initially determines whether data skew exists based on the number of records of candidate tables on each shard. If so, a target table that meets preset monitoring conditions is selected from the candidate tables, and a target operation instruction for the target table is sent to the candidate shard storing the target table. Based on the response time of each candidate shard, the monitoring shard is determined, and based on the monitoring results of slow SQL phenomena in each monitoring shard, data redistribution is performed if data skew is ultimately determined to exist. In this way, comprehensive and accurate monitoring of data in each shard of the distributed database can be achieved, precise detection of data skew can be made, and timely data redistribution can be performed, providing strong support for database operation and maintenance.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a data processing method provided in Embodiment 1 of the present invention;

[0022] Figure 2A This is a structural diagram of the distributed database automatic redistribution system provided in Embodiment 2 of the present invention;

[0023] Figure 2B This is a flowchart illustrating the automatic redistribution method provided in Embodiment 2 of the present invention;

[0024] Figure 3 This is a structural block diagram of a data processing device provided in Embodiment 3 of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of the electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," "target," "candidate," "alternative," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] It should be noted that uneven distribution of key-value pairs in a distributed database can lead to uneven distribution of query actions, causing some nodes to be overloaded. Therefore, online data redistribution is necessary. Furthermore, if the capacity and performance of sharded servers in a distributed network reach a bottleneck, the distributed database needs to be expanded. The automatic data redistribution detection scheme proposed in this invention, based on the number of sharded records and the time consumption of slow SQL queries, can monitor data distribution during expansion (such as adding data nodes to increase the processing capacity of the distributed database) by combining multi-dimensional information. If uneven data distribution is detected, i.e., data storage skew is determined, the data redistribution function of the distributed database is triggered to migrate and move data, adjust the data distribution, and achieve online data redistribution. The specific implementation process will be described in detail in subsequent embodiments.

[0029] Example 1

[0030] Figure 1 This is a flowchart of a data processing method provided in Embodiment 1 of the present invention. This embodiment can be applied to the automated operation and maintenance scenario of the Golden-DB distributed database. It is suitable for monitoring the data of each shard in the distributed database and redistributing the data when data skew is detected. This method can be executed by a data processing device, which can be implemented in hardware and / or software. The data processing device can be configured in an electronic device, such as in a distributed database, and executed by a distributed database automatic redistribution system. Figure 1 As shown, the data processing method includes:

[0031] S101. During the monitoring of each shard of the distributed database, the number of records of the candidate table on each shard is used to preliminarily determine whether there is data skew.

[0032] In this context, sharding refers to the shard servers corresponding to each shard in a distributed database. Each shard stores at least two candidate tables, which are used to store data in the distributed database. The number of records refers to the number of records stored in the candidate tables on the shard. Data skew refers to a significant deviation in the amount of data stored on different shards.

[0033] Optionally, you can enable tablespace monitoring and slow SQL (Structured Query Language) collection for the distributed database to obtain the number of records in candidate tables on each shard of the distributed database in real time.

[0034] Optionally, a preliminary determination of whether data skew exists includes: during the monitoring of each shard of the distributed database, determining the number of records of the candidate table on each shard; based on the number of records of the candidate table on each shard, if there is a target table in the candidate table that meets the preset monitoring conditions, then a preliminary determination of whether data skew exists is made.

[0035] For example, after determining the number of records in each shard of the candidate table, the percentage S of the standard deviation of the number of records in each shard relative to the standard deviation of the total number of records in the candidate table can be determined based on the following formula. i :

[0036]

[0037] Among them, S i The standard deviation is the percentage of the total standard deviation of the table's records, where n is the number of segments and x is the standard deviation of the total number of records. n Let be the total number of records in table i on the nth slice. Let be the average number of records in the i-th candidate table across all n partitions. Standard deviation (S) reflects the dispersion of data across different partitions. sum This represents the total number of rows in the table.

[0038] Optionally, after determining the percentage of the standard deviation of the number of records on each shard to the standard deviation of the total number of records in the candidate table, the relationship between the standard deviation percentage and the preset percentage threshold range can be determined. If the standard deviation percentage of the candidate table is not within the preset percentage threshold range, then the candidate table can be determined as the target table, that is, there is a target table in the candidate table that meets the preset monitoring conditions. At this time, it can be preliminarily determined that there is data skew.

[0039] It should be noted that when the dispersion of data across shards remains high, it may cause excessive pressure on individual shards, thereby reducing the overall performance of the distributed database. Using the standard deviation threshold to initially identify tables with data skew can effectively improve the efficiency of data skew detection.

[0040] S102. If so, then determine the target table that meets the preset monitoring conditions from the candidate table, and send the target operation instruction to the candidate shard that stores the target table.

[0041] The preset monitoring condition can be whether the standard deviation percentage of the candidate table is within a preset percentage threshold range. The target operation instruction can be a preset instruction to perform data query operations on each target table.

[0042] Optionally, the target table that meets the preset monitoring conditions is determined from the candidate table, including: during the monitoring of each shard of the distributed database, the number of records of the candidate table on each shard is periodically obtained, and the standard deviation percentage is calculated based on the number of records; the candidate table whose standard deviation percentage meets the preset monitoring conditions is determined as the target table.

[0043] Optionally, you can query the latest number of records on each shard by table, calculate the standard deviation of the number of records of the same candidate table on different shards, and the percentage of the standard deviation to the total number of records in the table, in order to determine the standard deviation percentage.

[0044] It should be noted that after initially determining the existence of data skew in step S101, the target table determined during step S101 can be directly used as the target table that meets the preset monitoring conditions, that is, the target table that meets the preset monitoring conditions is determined from the candidate table.

[0045] Optionally, after identifying the target table that meets the preset monitoring conditions from the candidate table, the target table can be cached and recorded so that the slow SQL phenomenon of the corresponding shard of the target table that is considered to have problems and needs to be monitored can be analyzed in the subsequent preliminary data skew judgment, and the situation of data skew can be further determined.

[0046] S103. Based on the response time of each candidate shard, determine the monitoring shard, and based on the monitoring results of slow SQL phenomena in each monitoring shard, redistribute the data if data skew is finally determined to exist.

[0047] In this context, a monitoring shard refers to a candidate shard whose response time exceeds a preset time threshold. Slow SQL refers to the phenomenon where a monitoring shard responds too slowly to the target operation command; specifically, this can be the monitoring shard's response time exceeding a preset time threshold.

[0048] Optionally, the monitoring shards are determined based on the response time of each candidate shard, including: determining the target shard with slow SQL behavior from the candidate shards based on the response time of each candidate shard, and determining the target table contained on the target shard as the monitoring table; and determining the target shard that stores the monitoring table as the monitoring shard corresponding to each monitoring table.

[0049] Optionally, based on the response time of each candidate shard, the target shard with slow SQL phenomenon is determined from the candidate shards, including: if the response time of a candidate shard is greater than a preset time threshold, then the candidate shard is determined to have slow SQL phenomenon.

[0050] It should be noted that there are many reasons for slow SQL queries, but slow SQL queries caused by data skew have a distinct characteristic: there are significant differences in query time across different shards. This characteristic is used to statistically analyze the query time of a table across each shard. At the same time, only the target tables that were initially identified as having problems and need to be monitored, i.e., the tables with records in the cache, are statistically analyzed.

[0051] Optionally, before redistributing data based on the monitoring results of slow SQL queries in each monitoring shard and in the event of data skew, it is possible to first monitor the slow SQL queries in each monitoring shard and generate monitoring results for each monitoring shard. These monitoring results include: sending monitoring operation commands to each monitoring shard at a preset period; determining whether a monitoring shard exhibits slow SQL queries based on its response time to the monitoring operation commands; and generating monitoring results for each monitoring shard based on the number of monitoring shards exhibiting slow SQL queries. Here, the monitoring operation commands refer to query operation commands to the monitoring tables.

[0052] Optionally, based on the monitoring results of slow SQL phenomena in each monitoring shard, if data skew is finally determined to exist, data redistribution is performed, including: based on the monitoring results of slow SQL phenomena in each monitoring shard, if some monitoring shards exhibit slow SQL phenomena, then data skew is finally determined to exist; trigger the data redistribution function of the distributed database to migrate and move data, adjust the data distribution, and achieve online data redistribution.

[0053] It should be noted that if all monitored shards exhibit slow SQL behavior, then the data redistribution operation need not be performed.

[0054] It should be noted that, given the dynamic nature of the data distribution status in distributed databases, a time dimension is added based on information such as the number of sharded records and the time consumption of slow SQL queries. This avoids the impact of sudden abnormal situations on the judgment results and improves the accuracy of data redistribution detection results.

[0055] It should be noted that after executing step S101, the table name of the target table and the standard deviation of the number of records in each shard are recorded in the cache so that the table name can be retrieved from the cache in step S103. Combined with the slow SQL information, the table that only has slow SQL on some shards and this phenomenon lasts for a certain period of time can be identified. If so, it can be determined that the data distribution of the table is uneven and data redistribution is required.

[0056] Optionally, after starting tablespace monitoring (i.e., after starting monitoring of each shard of the distributed database), the number of records in each shard can be periodically collected to calculate the standard deviation percentage. If data skew is initially determined to be possible, monitoring continues for a period of time to see if data skew still exists. If it is, even without slow SQL information, the distributed database is controlled to redistribute data. The reason for this phenomenon is that the business volume on each shard is relatively small in the recent period. As the business volume increases, the impact of uneven data distribution will become more apparent. Therefore, a more reliable approach is to resolve the problem of uneven data distribution in advance. The above implementation method can effectively solve the above problem.

[0057] The technical solution of this invention, during the monitoring of each shard of a distributed database, initially determines whether data skew exists based on the number of records of candidate tables on each shard. If so, a target table that meets preset monitoring conditions is selected from the candidate tables, and a target operation instruction for the target table is sent to the candidate shard storing the target table. Based on the response time of each candidate shard, the monitoring shard is determined, and based on the monitoring results of slow SQL phenomena in each monitoring shard, data redistribution is performed if data skew is ultimately determined to exist. In this way, comprehensive and accurate monitoring of data in each shard of the distributed database can be achieved, precise detection of data skew can be made, and timely data redistribution can be performed, providing strong support for database operation and maintenance.

[0058] Example 2

[0059] Figure 2A This is a structural diagram of the distributed database automatic redistribution system provided in Embodiment 2 of the present invention; Figure 2B This is a flowchart illustrating the automatic redistribution method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment proposes a preferred example for monitoring each shard of a distributed database to redistribute data in a timely manner when data skew occurs.

[0060] like Figure 2A As shown, the distributed database automatic redistribution system provided by the present invention may include: a collection module, an analysis module, a discrimination module, and an output module.

[0061] Specifically, the data collection module is used to collect basic information about the database, tablespace information, and slow SQL information (i.e., business SQL execution information).

[0062] Analysis module: Responsible for analyzing the basic information, tablespace information, and business SQL execution details of the source database, and statistics on table primary keys, unique keys, total number of rows for each shard, and query SQL execution time.

[0063] The discrimination module analyzes the data from the analysis module, identifies tables in each shard where there are significant differences in the total number of rows and the time consumed, and outputs the tables that require data redistribution.

[0064] The execution steps of the discrimination module can be divided into the following four steps:

[0065] (1) Count the total number of records in each table of each segment.

[0066] (2) Statistically analyze the slow-execution statements and their corresponding execution times in each segment.

[0067] (3) Filter out tables where there are significant differences in the total number of records and execution time for each segment.

[0068] (4) Continuously collect information from step (3). When the duration exceeds the upper limit, output an automatic data redistribution alarm output module: based on the statistics of information from multiple dimensions, output a report.

[0069] like Figure 2B As shown, after the acquisition module obtains basic information, tablespace information, and slow SQL information, it can further calculate the standard deviation of the number of sharded records to determine the standard version percentage. In combination with the slow query execution time (i.e. response time), data redistribution can be carried out when data skew is determined to occur.

[0070] Specifically, this invention relies on the indicator collection function to collect information from the target database, involving two preliminary steps:

[0071] 1) Enable tablespace monitoring and slow SQL collection;

[0072] 2) Analyze database tables, tablespaces, and slow SQL information. Statistically analyze primary keys, unique keys, foreign keys, tablespaces, and slow SQL data at the table level. Calculate the standard deviation of the total number of records in each shard and the mean of the slow SQL response time. If the variance exceeds a threshold and persists for a specified duration, trigger a data redistribution task. Details are as follows:

[0073] 1. Enable tablespace monitoring and slow SQL collection for distributed databases.

[0074] 2. Calculate the percentage of the standard deviation of the number of records in each segment relative to the total number of records in the table.

[0075] The standard deviation is calculated as follows: query the latest record count in each partition of the table, calculate the standard deviation of the record count in different partitions of the same table, and the percentage of the standard deviation to the total number of records in the table.

[0076] 3. Calculate the response time of query operations on each segment according to the table.

[0077] It should be noted that there are many reasons for slow SQL queries, but slow SQL queries caused by data skew have a distinct characteristic: there are significant differences in query time across different shards. This characteristic is used to statistically analyze the query time of a table across each shard. At the same time, only the target tables that were initially identified as having problems and need to be monitored, i.e., the tables with records in the cache, are statistically analyzed.

[0078] 4. Based on a comprehensive analysis of the standard deviation of the number of records on the shards in step 2 and the response time of the query operation in step 3, initiate automatic database redistribution.

[0079] In step 2, the table name and the standard deviation of the number of records in each shard are recorded in the cache. In step 3, the table name is retrieved from the cache in step 2, and combined with the slow SQL information, the table that only has slow SQL on some shards and this phenomenon lasts for a certain period of time is identified. If this is the case, it can be determined that the data distribution of the table is uneven and data redistribution is recommended.

[0080] In addition, there is another situation: the standard deviation of the number of records in each shard counted in step 2 has been stable for a period of time, but no slow SQL information is counted in step 3. In this case, there may also be a risk of data skew. Data redistribution is recommended. The possible reason is that the business volume on each shard is relatively small in the recent period. When the business volume increases, the impact of uneven data distribution will become more prominent. A more reliable way is to solve the problem of uneven data distribution in advance.

[0081] In other words, after starting tablespace monitoring, the percentage of records in each shard is periodically collected. If it is initially determined that there may be data skew, monitoring continues for a period of time to see if data skew still exists. If so, even if there is no slow SQL information, the distributed database is controlled to redistribute data.

[0082] 5. When the data dispersion across shards remains high, it may cause excessive pressure on individual shards, thereby reducing the overall performance of the distributed database. Tables with data skew can be initially identified by using the standard deviation threshold.

[0083] The technical solution of this invention has the following beneficial effects: Timeliness: It calculates the data distribution according to specified rules and monitors the distribution data within a certain time period, which is more efficient than manual judgment. High accuracy: This invention comprehensively considers information such as the number of records in each shard and the time consumption of slow SQL queries, and adds a time dimension, which further increases the stability of the judgment, avoids the impact of sudden abnormal situations on the judgment results, and improves the accuracy of detection.

[0084] Example 3

[0085] Figure 3This is a structural block diagram of a data processing device provided in Embodiment 3 of the present invention. This embodiment is applicable to monitoring data in each shard of a distributed database and redistributing data when data skew is detected. The data processing device can be implemented in hardware and / or software and configured in a device with data processing capabilities, such as a distributed database, and executed by an automatic redistribution system for the distributed database. Figure 3 As shown, the device specifically includes:

[0086] The determination module 301 is used to preliminarily determine whether there is data skew based on the number of records of the candidate table on each shard during the monitoring of each shard of the distributed database.

[0087] The sending module 302 is used to determine the target table that meets the preset monitoring conditions from the candidate table if the conditions are met, and send the target operation instruction to the candidate segment that stores the target table.

[0088] The redistribution module 303 is used to determine the monitoring shards based on the response time of each candidate shard, and to redistribute the data based on the monitoring results of slow SQL phenomena of each monitoring shard, if it is finally determined that there is data skew.

[0089] The technical solution of this invention, during the monitoring of each shard of a distributed database, initially determines whether data skew exists based on the number of records of candidate tables on each shard. If so, a target table that meets preset monitoring conditions is selected from the candidate tables, and a target operation instruction for the target table is sent to the candidate shard storing the target table. Based on the response time of each candidate shard, the monitoring shard is determined, and based on the monitoring results of slow SQL phenomena in each monitoring shard, data redistribution is performed if data skew is ultimately determined to exist. In this way, comprehensive and accurate monitoring of data in each shard of the distributed database can be achieved, precise detection of data skew can be made, and timely data redistribution can be performed, providing strong support for database operation and maintenance.

[0090] Furthermore, module 301 is specifically used for:

[0091] During the monitoring of each shard of the distributed database, the number of records of the candidate table on each shard is counted to determine the total number of records of the candidate table.

[0092] Based on the number of records in each shard of the candidate table, if there is a target table in the candidate table that meets the preset monitoring conditions, it is preliminarily determined that there is data skew.

[0093] Furthermore, the sending module 302 is specifically used for:

[0094] During the monitoring of each shard of the distributed database, the number of records of the candidate table on each shard is periodically obtained, and the standard deviation percentage is calculated based on the number of records.

[0095] Candidate tables whose standard deviation percentage meets the preset monitoring conditions are selected as target tables.

[0096] Furthermore, the redistribution module 303 may include:

[0097] The first determining unit is used to determine the target shard with slow SQL phenomenon from the candidate shards based on the response time of each candidate shard, and to determine the target table contained in the target shard as the monitoring table.

[0098] The second determining unit is used to determine the target fragment containing the monitoring table as the monitoring fragment corresponding to each monitoring table.

[0099] Furthermore, the first determining unit is specifically used for:

[0100] If the response time of a candidate shard is greater than a preset time threshold, then the candidate shard is determined to have slow SQL behavior.

[0101] Furthermore, the redistribution module 303 is specifically used for:

[0102] Based on a preset cycle, monitoring operation instructions for the monitoring table are sent to each monitoring segment.

[0103] Based on the response time of each monitoring segment to monitoring operation commands, determine whether there is slow SQL in each monitoring segment;

[0104] Based on the number of monitoring shards exhibiting slow SQL behavior, generate monitoring results for slow SQL behavior in each monitoring shard.

[0105] Furthermore, the redistribution module 303 is also used for:

[0106] Based on the monitoring results of slow SQL phenomena in each monitoring shard, if some monitoring shards exhibit slow SQL phenomena, then data skew is ultimately determined to exist.

[0107] Trigger the data redistribution function of the distributed database to migrate and move data, adjust data distribution, and achieve online data redistribution.

[0108] Example 4

[0109] Figure 4 This is a schematic diagram of the structure of the electronic device provided in Embodiment 4 of the present invention. Figure 4A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0110] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0111] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods.

[0113] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

[0114] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0119] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0120] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized by, Comprise: In the process of monitoring each shard of the distributed database, whether there is data skew is preliminarily determined according to the number of records of the candidate table on each shard; If yes, the target table meeting the preset monitoring condition is determined from the candidate table, and a target operation instruction for the target table is sent to the candidate shard storing the target table; According to the response time of each candidate shard, the monitoring shard is determined, and in the case of finally determining that there is data skew, data redistribution is performed according to the monitoring result of the slow SQL phenomenon of each monitoring shard; The preliminary determination of whether there is data skew comprises: In the process of monitoring each shard of the distributed database, the number of records of the candidate table on each shard is determined; According to the number of records of the candidate table on each shard, if there is a target table meeting the preset monitoring condition in the candidate table, it is preliminarily determined that there is data skew; The target table meeting the preset monitoring condition is determined from the candidate table, comprising: In the process of monitoring each shard of the distributed database, the number of records of the candidate table on each shard is periodically obtained, and the standard deviation percentage is calculated according to the number of records; The candidate table whose standard deviation percentage meets the preset monitoring condition is determined as the target table; According to the monitoring result of the slow SQL phenomenon of each monitoring shard, if there is a part of monitoring shards that appear slow SQL phenomenon, it is finally determined that there is data skew; Trigger the data redistribution function of the distributed database to move the data and adjust the data distribution to realize online data redistribution. According to the response time of each candidate shard, the monitoring shard is determined, comprising:

2. The method of claim 1, wherein, According to the response time of each candidate shard, the target shard with slow SQL phenomenon is determined from the candidate shard, and the target table contained in the target shard is determined as the monitoring table; The target shard storing the monitoring table is determined as the monitoring shard corresponding to each monitoring table. According to the response time of each candidate shard, the target shard with slow SQL phenomenon is determined from the candidate shard, comprising:

3. The method of claim 2, wherein, If the response time of the candidate shard is greater than the preset time threshold, it is determined that the candidate shard has slow SQL phenomenon. The monitoring result of the slow SQL phenomenon of each monitoring shard is generated, comprising:

4. The method of claim 1, wherein, Based on the preset period, a monitoring operation instruction for the monitoring table is sent to each monitoring shard; According to the response time of each monitoring shard to the monitoring operation instruction, whether the monitoring shard has slow SQL phenomenon is determined respectively; According to the number of monitoring shards with slow SQL phenomenon, the monitoring result of the slow SQL phenomenon of each monitoring shard is generated. Comprise:

5. A data processing apparatus, characterized by, A determination module, configured to preliminarily determine whether there is data skew according to the number of records of the candidate table on each shard in the process of monitoring each shard of the distributed database; A sending module, configured to if yes, determine the target table meeting the preset monitoring condition from the candidate table, and send a target operation instruction for the target table to the candidate shard storing the target table; ​ The redistribution module is configured to determine the monitoring shards according to the response times of the candidate shards, and perform data redistribution in a case where it is finally determined that data skew exists according to the monitoring results of the slow SQL phenomenon of the monitoring shards. The determining module is specifically configured to: In the process of monitoring each shard of the distributed database, the number of records of the candidate table on each shard is determined; According to the number of records of the candidate table on each shard, if there is a target table in the candidate table that meets the preset monitoring condition, it is preliminarily determined that data skew exists; The sending module is specifically configured to: In the process of monitoring each shard of the distributed database, the number of records of the candidate table on each shard is periodically obtained, and the standard deviation percentage is calculated according to the number of records; The candidate table whose standard deviation percentage meets the preset monitoring condition is determined as the target table; The redistribution module is further configured to: According to the monitoring results of the slow SQL phenomenon of each monitoring shard, if there is a part of the monitoring shards that appear the slow SQL phenomenon, it is finally determined that data skew exists; Triggering the data redistribution function of the distributed database to perform data migration and adjust data distribution, so as to realize online data redistribution.

6. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data processing method of any one of claims 1-4.

7. A computer readable storage medium characterized by The computer readable storage medium stores computer instructions for enabling the processor to execute the data processing method of any one of claims 1-4 when executed.

Images