Data skew early warning method, device, equipment, medium and program product

Abstract

The invention discloses a data skew early warning method, device and equipment, a medium and a program product. The invention relates to a big data technology and a distributed technology, and is also suitable for financial science and technology scenes. The method comprises the following steps: acquiring data distribution information of nodes in a distributed database, and determining a tilt table in the distributed database according to the data distribution information of the nodes; database statements corresponding to the tilt table are obtained, tilt early warning information is determined according to data distribution information of target fields in the database statements, and the database statements comprise database operation statements and database table building statements; and executing data inclination early warning according to the inclination early warning information. According to the technical scheme of the invention, the data skew is automatically detected based on the data distribution information, and early warning is carried out, so that the time consumption for discovering the data skew problem can be shortened, the time consumption from occurrence to solution of the data skew problem can be shortened, and the database performance and the service processing time efficiency are ensured.

Description

Technical Field

[0001] The embodiments of the present invention relate to big data technology and distributed technology, and are also applicable to financial technology scenarios, and in particular to a data tilt warning method, device, equipment, medium and program product. Background Art

[0002] With the advent of the big data era, the amount of data is showing an explosive growth trend. As an efficient and scalable data storage and processing technology, distributed databases are increasingly widely used in the field of big data. Distributed databases achieve efficient management and processing of large-scale data sets by distributing and storing data on multiple nodes.

[0003] However, in the actual use of distributed databases, data skew has become a common pain point. Data skew refers to the uneven distribution of data on different nodes in a distributed system, resulting in the number, computing load or I / O pressure of some nodes being significantly higher than other nodes. This uneven data distribution can lead to a decline in system performance. Currently, in distributed databases, data skew problems are usually discovered through manual troubleshooting. It takes a long time from the occurrence of data skew problems to the discovery of the problem, which affects database performance and business processing timeliness. Summary of the invention

[0004] The embodiments of the present invention provide a data skew warning method, device, equipment, medium and program product, which can identify data skew and issue a warning, effectively shorten the time spent on discovering data skew problems, and ensure database performance and business processing timeliness.

[0005] In a first aspect, an embodiment of the present invention provides a data tilt warning method, comprising:

[0006] Acquire data distribution information of nodes in a distributed database, and determine a tilted table in the distributed database according to the data distribution information of the nodes;

[0007] Obtaining a database statement corresponding to the tilted table, and determining tilt warning information according to data distribution information of a target field in the database statement, wherein the database statement includes a database operation statement and a database table creation statement;

[0008] A data tilt warning is performed according to the tilt warning information.

[0009] In a second aspect, an embodiment of the present invention further provides a data tilt warning device, the device comprising:

[0010] A tilt table determination module, used to obtain data distribution information of nodes in a distributed database, and determine a tilt table in the distributed database according to the data distribution information of the nodes;

[0011] A tilt warning information determination module, used to obtain a database statement corresponding to the tilt table, and determine the tilt warning information according to the data distribution information of the target field in the database statement, wherein the database statement includes a database operation statement and a database table creation statement;

[0012] The tilt warning module is used to perform data tilt warning according to the tilt warning information.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a data skew warning method as described in any one of the embodiments of the present invention is implemented.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data skew warning method as described in any one of the embodiments of the present invention.

[0015] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a data skew warning method as described in any one of the embodiments of the present invention.

[0016] The embodiment of the present invention provides a data tilt warning scheme, which determines the tilt table through the data distribution information of the nodes in the distributed database, then obtains the database statement corresponding to the tilt table, determines the tilt warning information according to the data distribution information of the target field in the database statement, and executes the data tilt warning according to the tilt warning information. The technical scheme of the embodiment of the present invention automatically detects and identifies data tilt based on data distribution information, and issues a warning, which can shorten the time spent on discovering data tilt problems, greatly reduce the cost of manual intervention, and ensure database performance and business processing timeliness. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of a process flow of a data tilt early warning method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a flow chart of another data tilt warning method provided by an embodiment of the present invention;

[0020] Figure 3 A framework diagram of a data tilt early warning system provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the structure of a data tilt warning device provided by an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0024] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0025] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0028] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.

[0029] Figure 1A flow chart of a data tilt warning method provided by an embodiment of the present invention. The present invention is applicable to big data technology, distributed technology, and also to financial technology scenarios. The device can be implemented in the form of software and / or hardware, and optionally, implemented by an electronic device, which can be a mobile terminal, a PC, or a server.

[0030] like Figure 1 As shown, the method includes:

[0031] S110 , obtaining data distribution information of nodes in a distributed database, and determining a tilted table in the distributed database according to the data distribution information of the nodes.

[0032] Among them, a distributed database refers to data distributed on different nodes of a computing network, which together form a logically unified database system. A distributed database system can improve data access efficiency, enhance the reliability and scalability of the database system, and support large-scale data processing and complex query operations. Distributed databases not only provide powerful data storage capabilities, but also support flexible data processing and analysis capabilities, providing strong support for big data applications.

[0033] Data distribution information represents the information of data distributed on the nodes of a distributed database. For example, data distribution information includes node load information and / or table data volume, etc. Among them, node load information refers to the amount of data and requests that a single node needs to process within a set time. For example, node load includes at least one of data storage load, computing load, and I / O load. Table data volume is the amount of data in a data table, which represents the storage space occupied by the data table.

[0034] A tilted table is a data table with data tilt. The amount of data in a tilted table is higher than that of other data tables on the same node. For example, if node A includes data tables a, b, and c, and the amount of data in data table b is much larger than that of the other two data tables, then data table b is determined to be a tilted table.

[0035] Exemplarily, obtaining data distribution information of a node in a distributed database includes: obtaining load information of a node in the distributed database as data distribution information of the node. In order to reduce consumption of database computing resources, the nodes in the distributed database may be monitored. For example, the load information of the nodes of the distributed database may be detected to determine whether the loads of the nodes are relatively balanced. By monitoring the load information of each node, it may be determined based on the load information whether the node has a performance bottleneck, thereby realizing the use of a coarse-grained approach to identify data skew.

[0036] For example, the data storage load of each node in the distributed database is obtained through a database monitoring tool, and the data distribution information of the nodes in the distributed database is determined according to the data storage load.

[0037] Exemplarily, determining the tilted table in the distributed database according to the data distribution information of the node includes: determining a load reference value according to the load information of the node in the distributed database, and determining a load deviation between the load information of the node and the load reference value. Determining a tilted node according to the node whose load deviation exceeds a preset deviation threshold. Obtaining the data volume of the data table on the tilted node, and determining the tilted table on the tilted node according to the data volume.

[0038] The load reference value represents the overall load information of all nodes in the distributed database. The load reference value is determined according to the load information of each node in the distributed database. For example, the load reference value may include the load average value of the node. Alternatively, the load reference value may also include the weighted average value of the node. The load deviation represents the difference between the load information of a single node and the load reference value.

[0039] For example, according to the data storage load of each node in the distributed database, the data load average value is calculated. The data load average value is used as the load reference value. The difference between the data storage load of each node and the data load average value is calculated, and the division result of the difference and the data average value is calculated as the load deviation between the load information of the node and the load reference value. If the load deviation corresponding to the node exceeds the preset deviation threshold, the node is regarded as a tilted node.

[0040] Furthermore, the size of the storage space occupied by the data table on the tilted node is obtained as the data volume of the data table. The data volumes of the data tables are compared to determine the tilted table where data tilt occurs. For example, if the tilted node includes data table a, data table b, and data table c, and the data volume of data table a is greater than the data volume of data table b and the data volume of data table c, and the deviation of the data volume exceeds a preset deviation threshold, then data table a is determined to be a tilted table.

[0041] The embodiment of the present invention determines the tilted node through the load information of the node, and then performs data volume statistics on the data table on the tilted node, and then determines the tilted table according to the data volume statistical result. It can identify data tilt in a fine-grained manner and reduce the consumption of database computing resources.

[0042] S120. Obtain a database statement corresponding to the tilted table, and determine tilt warning information according to data distribution information of a target field in the database statement, wherein the database statement includes a database operation statement and a database table creation statement.

[0043] The database statement represents a statement for operating data. For example, the database statement includes a database operation statement and a database table creation statement. The database operation statement includes a database query statement. The target field is a key value field in the database statement. For example, the target field includes a distribution key, a primary key of a table, or a combination of one or more other commonly used fields.

[0044] The data distribution information of the target field represents the distribution of the data corresponding to the target field on each node. For example, the distribution key is used to determine the key attributes of the data distribution and storage information on different nodes. Assuming that the distribution key is the object ID, the object data can be segmented and stored on multiple nodes according to the object ID, and the amount of data corresponding to the distribution key on each node can be determined based on the object ID.

[0045] The tilt warning information represents information used for data tilt warning. For example, the tilt warning information includes detailed information of the tilt table. Optionally, the tilt warning information includes at least one of the tilt table name, tilt field, tilt degree, and tilt reason.

[0046] Exemplarily, the hot data in the query operation is determined by the historical query statements and / or historical table creation statements corresponding to the tilted table. Among them, the hot data generally includes the distribution key, the primary key of the table and other commonly used field data. According to the hot data, the frequently accessed fields are found, and the cause of the tilt is determined according to the data distribution associated with the frequently accessed fields. Optionally, by analyzing the historical query statements and / or historical table creation statements corresponding to the tilted table, the association relationship between the fields is obtained, and then the data dependency that may cause the data tilt is determined. The cause of the tilt is determined according to the data dependency. Among them, the data dependency is that the amount of data of certain key values ​​is much larger than that of other key values, resulting in unbalanced load during data processing such as data aggregation or data connection, thereby affecting database performance and resource utilization. After determining the cause of the tilt, the tilt warning information is determined according to the cause of the tilt.

[0047] Determine the data volume of each data table on the tilted node according to the size of the occupied storage space. Determine the data table whose first data volume deviation exceeds the set threshold according to the data volume of each data table, and determine the data table whose first data volume deviation exceeds the set threshold as a tilted table. Determine the tilt warning information according to the table name of the tilted table. Among them, the first data volume deviation represents the deviation information of the data volume between each data table.

[0048] The tilt degree is determined according to data distribution information of a target field in a database statement associated with the tilt table, and tilt warning information is determined according to the tilt degree.

[0049] S130. Execute data tilt warning according to the tilt warning information.

[0050] Among them, data skew warning refers to the warning operation for the data skew problem of the distributed database. For example, the data skew warning includes sending a data skew notification. The relevant operation and maintenance personnel can be notified by SMS or email, so as to deal with the database skew problem in a timely manner. Optionally, the data skew notification may include data skew information. Optionally, the data skew notification may also include a reference solution, etc. The reference solution is the solution suggestion information for the data skew problem. The reference solution is determined according to the cause of the skew.

[0051] Exemplarily, performing data tilt warning according to the tilt warning information includes: if the tilt in the tilt warning information exceeds a set tilt threshold, determining that the distributed database has a data tilt problem, saving the tilt warning information corresponding to the data tilt, and sending a data tilt notification according to the tilt warning information.

[0052] Set a reasonable skew threshold based on business needs and historical data. There may be a small degree of data skew in the nodes and / or data tables in the distributed database. For example, different nodes have different node loads. Different data tables have different amounts of data. When the degree of data skew exceeds the set skew threshold, it is determined that the distributed database has a data skew problem. Based on the comparison result of the skew degree of the skewed table with the set skew threshold, it is possible to monitor whether the distributed database has a data skew problem.

[0053] If data skew occurs, the detailed information of the data skew is recorded as a basis for subsequent analysis and resolution of the data skew problem. In addition, data skew notifications are sent to relevant operation and maintenance personnel via SMS or email to provide timely warnings of data skew problems.

[0054] The embodiment of the present invention automatically identifies data skew problems by setting a skew threshold, and when a data skew problem occurs, records detailed information about the data skew and sends a data skew notification, thereby enabling timely discovery and early warning of data skew problems, greatly shortening the time from the occurrence of the data skew problem to the discovery of the data skew problem by operation and maintenance personnel, making it easier for operation and maintenance personnel to solve the data skew problem in a timely manner.

[0055] Optionally, sending the data tilt notification according to the tilt warning information includes: determining a reference solution according to a tilt cause in the tilt warning information, and sending the data tilt notification according to the reference solution and the tilt warning information.

[0056] The reference solutions include: data migration, data partitioning, index optimization, and query optimization. Data migration is to migrate data from high-load nodes to low-load nodes to balance the load of each node. Data partitioning is to reasonably divide data partitions and optimize data distribution according to data characteristics and query requirements. Index optimization is to create indexes for frequently queried data to improve query efficiency. Query optimization refers to adjusting query operations, reducing cross-node queries, and reducing query complexity. The reference solution is pre-configured, and a mapping relationship between the cause of the tilt and the reference solution can be established. Therefore, the corresponding reference solution is selected according to the cause of the tilt.

[0057] For example, if the reason for the skew is that a suitable distribution key cannot be found, the reference solution is data migration. That is, the load of each node is balanced by migrating the high-load node to the low-load node. If the reason for the skew is the wrong distribution key selection, the reference solution is data partitioning. That is, according to the data characteristics and query requirements, the data partitions are reasonably divided to optimize the data distribution.

[0058] The embodiment of the present invention determines a reference solution based on the cause of the skew, and sends the reference solution to the operation and maintenance personnel through a data skew notification, thereby displaying multiple solutions to the data skew problem for reference by the operation and maintenance personnel, simplifying the difficulty of solving the data skew problem and facilitating shortening the time spent on solving the data skew problem.

[0059] The technical solution of the embodiment of the present invention determines the tilt table through the data distribution information of the nodes in the distributed database, then obtains the database statement corresponding to the tilt table, determines the tilt warning information according to the data distribution information of the target field in the database statement, and executes the data tilt warning according to the tilt warning information. The technical solution of the embodiment of the present invention automatically detects and identifies data tilt based on data distribution information, and issues a warning, which can shorten the time spent on discovering data tilt problems, greatly reduce the cost of manual intervention, and ensure database performance and business processing timeliness.

[0060] Figure 2 A flowchart of another data tilt warning method provided by an embodiment of the present invention. Based on the above embodiments, the embodiment of the present invention specifically defines determining tilt warning information according to data distribution information of a target field in the database statement. The target field represents a combination of key value fields.

[0061] like Figure 2 As shown, the method includes:

[0062] S210 , obtaining data distribution information of nodes in a distributed database, and determining a tilted table in the distributed database according to the data distribution information of the nodes.

[0063] S220: Obtain database statements corresponding to the tilted table, wherein the database statements include database operation statements and database table creation statements.

[0064] S230. Determine a target node corresponding to the target field according to the database statement, wherein the target node represents the node storing the data record corresponding to the target field.

[0065] Exemplarily, according to the database statement, the data distribution corresponding to each key value field in the combination of key value fields is determined. The data distribution represents the target node where the data is located. The data table on each node can be queried according to the distribution key and / or the primary key to obtain a data table containing the above distribution key and / or the primary key. For example, according to the distribution key ss, the field name of the data table on each node is queried to obtain a data table containing the ss field. The node of the data table including the ss field is used as the target node.

[0066] S240. Determine data distribution information of the target field according to the data amount of the target field on the target node.

[0067] Exemplarily, the data distribution information of the target field is determined according to the data amount of the data table containing the target field on each target node. For example, assuming that the target field is ss, and the nodes where the data table containing the target field is located are node A, node B, and node C, then the data distribution information of the target field includes: the data amount of the data table containing the ss field on node A, the amount of the data table containing the ss field on node B, and the data amount of the data table containing the ss field on node C.

[0068] S250. Determine the tilted field, tilt degree and tilt reason according to the data distribution information of the target field.

[0069] Exemplarily, the tilted field is determined according to the data volume of each key value field in the combination of the key value fields. The tilt degree is determined according to the deviation between the data volume of the tilted field and the data volume of the combination of the key value fields. The tilt cause is determined according to the data volume of the combination of the key value fields.

[0070] For example, the data volume of each key value field in the combination of key value fields is compared to determine the tilted field where data tilt occurs. Optionally, a second data volume deviation is determined according to the data volume of each key value field in the combination of key value fields, and the tilted field is determined according to the second data volume deviation. The second data volume deviation represents the deviation information of the data volume corresponding to each key value field. For example, a deviation threshold can be pre-set, and a key value field whose second data volume deviation exceeds the preset deviation threshold is determined as a tilted field.

[0071] The average data volume is calculated based on the data volume of each key value field in the combination of key value fields. The data volume difference between the data volume of each key value field and the data volume average is calculated. The quotient of the data volume difference corresponding to each key value field and the data volume average is calculated as the inclination of each key value field.

[0072] According to the data amount of each key-value field in the combination of key-value fields, the data distribution of each key-value field is analyzed, and then the cause of the skew is determined according to the data distribution.

[0073] The embodiment of the present invention determines the tilted field, tilt degree and tilt reason according to the data volume of each key value field, thereby automatically generating detailed information on the data tilt problem, which can greatly reduce manual intervention.

[0074] S260: Generate the tilt warning information according to the table name, tilt field, tilt degree and tilt reason of the tilt table.

[0075] Since the tilt warning information includes the table name, tilt field, tilt degree and tilt reason, the table name, tilt field, tilt degree and tilt reason can be spliced ​​according to the set organization format to obtain the tilt warning information.

[0076] S270: Execute data tilt warning according to the tilt warning information.

[0077] The technical solution of the embodiment of the present invention analyzes the amount of data distributed on the nodes of the distributed database of the target field according to the database statement corresponding to the tilted table, and determines the data distribution information of the target field according to the amount of data distributed on the node. Then, the tilted field, tilt degree and tilt reason are determined according to the data distribution information of the target field, and then, the tilt warning information is generated according to the table name, tilted field, tilt degree and tilt reason of the tilted table, and the data tilt warning is executed according to the tilt warning information. Through the technical solution of the embodiment of the present invention, the target field is determined by monitoring the database statement corresponding to the tilted table, and the tilt warning information is determined according to the distribution of the data amount of the target field on each node, so as to realize real-time monitoring of the distributed database, which can effectively solve the problem of system performance degradation caused by data tilt in the distributed database, reduce maintenance costs, and improve the stability and reliability of business implementation.

[0078] Figure 3 The following is a framework diagram of a data tilt warning system provided by an embodiment of the present invention. Figure 3As shown, the data tilt early warning system 310 is respectively connected to the distributed database system 320 and the metadata system 330. Each node in the distributed database system 320 is used to store data. The metadata system 330 is used to store database object information. The metadata system 330 is an essential component of the distributed database system 320. The data tilt early warning system 310 includes a data exploration module 311, a data analysis module 312, an early warning feedback module 313 and a problem handling module 314. The data exploration module 311 monitors the data distribution of each node in real time. For example, the data exploration module 311 monitors the load of each node in real time and analyzes the tilted nodes that may have data tilt. Then, the data volume of the data tables distributed on the tilted nodes is counted, and the data tables with data tilt, i.e., tilted tables, are determined according to the data volume of each data table. The information of the tilted table is input into the data analysis module 312, and the historical database operation statements and historical table creation statements related to the tilted table are analyzed by the data analysis module 312, and the key value field combination is counted. The tilt degree, tilt field and tilt reason are determined according to the amount of data distributed on each node of each key value field in the key value field combination. The data analysis module 312 sends the table name, tilt degree, tilt field and tilt reason of the tilted table to the early warning feedback module 313. The early warning feedback module 313 sends a data tilt notification to the operation and maintenance personnel based on the table name, tilt degree, tilt field and tilt reason of the tilted table. The problem handling module 314 determines a reference solution based on the tilt reason of the tilted table, and the reference solution can be carried in the data tilt notification sent to the operation and maintenance personnel.

[0079] Figure 4 A schematic diagram of the structure of a data tilt warning device provided by an embodiment of the present invention. The device can be implemented in the form of software and / or hardware, and optionally, implemented by an electronic device, which can be a mobile terminal, a PC or a server.

[0080] like Figure 4 Said device comprises:

[0081] A tilt table determination module 410 is used to obtain data distribution information of a node in a distributed database, and determine a tilt table in the distributed database according to the data distribution information of the node;

[0082] The tilt warning information determination module 420 is used to obtain the database statement corresponding to the tilt table, and determine the tilt warning information according to the data distribution information of the target field in the database statement, wherein the database statement includes a database operation statement and a database table creation statement;

[0083] The tilt warning module 430 is used to perform data tilt warning according to the tilt warning information.

[0084] Optionally, the tilt table determination module 410 is specifically configured to:

[0085] The load information of the nodes in the distributed database is obtained as the data distribution information of the nodes.

[0086] Optionally, the tilt table determination module 410 is further configured to:

[0087] Determine a load reference value according to the load information of the node in the distributed database, and determine a load deviation between the load information of the node and the load reference value;

[0088] Determine a tilted node according to the node whose load deviation exceeds a preset deviation threshold;

[0089] The data volume of the data table on the inclined node is obtained, and the inclined table on the inclined node is determined according to the data volume.

[0090] Optionally, the tilt warning information determination module 420 is specifically used for:

[0091] Determine a target node corresponding to the target field according to the database statement, wherein the target node represents the node storing the data record corresponding to the target field;

[0092] Determining data distribution information of the target field according to the data amount of the target field on the target node;

[0093] Determine the tilted field, tilt degree and tilt reason according to the data distribution information of the target field;

[0094] The tilt warning information is generated according to the table name, tilt field, tilt degree and tilt reason of the tilt table.

[0095] Optionally, the target field represents a combination of key-value fields;

[0096] The step of determining the tilted field, tilt degree and tilt reason according to the data distribution information of the target field includes:

[0097] Determining the tilted field according to the data amount of each key value field in the combination of the key value fields;

[0098] Determining the inclination according to a deviation between the amount of data of the inclination field and the amount of data of the combination of the key value fields;

[0099] The tilt cause is determined according to the data volume of the combination of the key value fields.

[0100] Optionally, the tilt warning module 430 is specifically used for:

[0101] If the tilt degree in the tilt warning information exceeds a set tilt threshold, it is determined that a data tilt problem occurs in the distributed database;

[0102] The tilt warning information corresponding to the data tilt is saved, and a data tilt notification is sent according to the tilt warning information.

[0103] Optionally, sending a data tilt notification according to the tilt warning information includes:

[0104] A reference solution is determined according to the tilt cause in the tilt warning information, and a data tilt notification is sent according to the reference solution and the tilt warning information.

[0105] The data tilt warning device provided in the embodiment of the present invention can execute the data tilt warning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0106] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, and other suitable computers. The components shown herein, their connections and relationships, and their functions are only for example and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0107] like Figure 5 As shown, the electronic device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 to the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0108] A number of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The processor 51 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 executes the various methods and processes described above, such as a data tilt warning method.

[0110] In some embodiments, a data tilt warning method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to perform a data tilt warning method in any other appropriate manner (e.g., by means of firmware).

[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable rule-based decision-making device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0113] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0116] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0117] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data skew warning method provided in any embodiment of the present application.

[0118] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0119] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data tilt early warning method, characterized in that: include: Acquire data distribution information of nodes in a distributed database, and determine a tilted table in the distributed database according to the data distribution information of the nodes; Obtaining a database statement corresponding to the tilted table, and determining tilt warning information according to data distribution information of a target field in the database statement, wherein the database statement includes a database operation statement and a database table creation statement; A data tilt warning is performed according to the tilt warning information.

2. The method according to claim 1, characterized in that: The obtaining of data distribution information of nodes in a distributed database includes: The load information of the nodes in the distributed database is obtained as the data distribution information of the nodes.

3. The method according to claim 2, characterized in that The determining the tilted table in the distributed database according to the data distribution information of the node includes: Determine a load reference value according to the load information of the node in the distributed database, and determine a load deviation between the load information of the node and the load reference value; Determine a tilted node according to the node whose load deviation exceeds a preset deviation threshold; The data volume of the data table on the inclined node is obtained, and the inclined table on the inclined node is determined according to the data volume.

4. The method according to claim 1, characterized in that The determining the tilt warning information according to the data distribution information of the target field in the database statement includes: Determine a target node corresponding to the target field according to the database statement, wherein the target node represents the node storing the data record corresponding to the target field; Determining data distribution information of the target field according to the data amount of the target field on the target node; Determine the tilted field, tilt degree and tilt reason according to the data distribution information of the target field; The tilt warning information is generated according to the table name, tilt field, tilt degree and tilt reason of the tilt table.

5. The method according to claim 4, characterized in that The target field represents a combination of key-value fields; The step of determining the tilted field, tilt degree and tilt reason according to the data distribution information of the target field includes: Determining the tilted field according to the data amount of each key value field in the combination of the key value fields; Determining the inclination according to a deviation between the amount of data of the inclination field and the amount of data of the combination of the key value fields; The tilt cause is determined according to the data volume of the combination of the key value fields.

6. The method according to claim 1, characterized in that The performing data tilt warning according to the tilt warning information includes: If the tilt degree in the tilt warning information exceeds a set tilt threshold, it is determined that a data tilt problem occurs in the distributed database; The tilt warning information corresponding to the data tilt is saved, and a data tilt notification is sent according to the tilt warning information.

7. The method according to claim 6, characterized in that The sending of a data tilt notification according to the tilt warning information includes: A reference solution is determined according to the tilt cause in the tilt warning information, and a data tilt notification is sent according to the reference solution and the tilt warning information.

8. A data tilt warning device, characterized in that: include: A tilt table determination module, used to obtain data distribution information of nodes in a distributed database, and determine a tilt table in the distributed database according to the data distribution information of the nodes; A tilt warning information determination module, used to obtain a database statement corresponding to the tilt table, and determine the tilt warning information according to the data distribution information of the target field in the database statement, wherein the database statement includes a database operation statement and a database table creation statement; The tilt warning module is used to perform data tilt warning according to the tilt warning information.

9. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data tilt warning method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data tilt warning method as described in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data tilt warning method according to any one of claims 1 to 7.

Images