Method for improving IO performance of parallel data warehouse using distributed virtual storage

The central control module calculates the IO performance evaluation parameters of the database and adjusts the memory configuration of the database and server, which solves the problem of insufficient memory planning for distributed storage devices and improves the IO performance and operation speed of parallel data warehouses.

CN117235189BActive Publication Date: 2024-11-22SHENZHEN JINXINQUAN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310162137.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-02-15
Publication Date
2024-11-22
Estimated Expiration
2043-02-15

AI Technical Summary

Technical Problem

The prior art does not perform memory planning for distributed storage devices, which affects the IO performance and operating speed of parallel data warehouses.

Method used

The central control module calculates the database operating status based on the database's IO performance evaluation parameters, adjusts the memory configuration of the database and server, including data migration and reserved memory, to optimize IO performance.

Benefits of technology

It effectively improves the IO performance and operation speed of parallel data warehouses, and realizes the rational allocation of servers and database resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117235189B_ABST
    Figure CN117235189B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of storage virtualization, and in particular to a method for improving the IO performance of a parallel data warehouse by using distributed virtual storage, comprising: step S1, a central control module determines the operation state of a database according to a calculated database IO performance evaluation parameter P; step S2, adjusting the preset evaluation parameter of the database according to the number of bytes of data read by the database at a single time; step S3, adjusting the overflow data of the database to an affiliated database of the database according to P, and adjusting the reserved memory of a single database according to the total amount of newly added affiliated databases and the average evaluation parameter value of several databases in a single server; step S4, adjusting the data in the server according to the re-acquired average evaluation parameter value of the single server; the present invention further improves the operation speed of the parallel distributed data warehouse while effectively improving the IO performance of the parallel data warehouse.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of storage virtualization, and in particular to a method for improving the IO performance of a parallel data warehouse by using distributed virtual storage. Background Art

[0002] With the development and application of Internet technology, a large amount of data that needs to be stored will be generated in the process of Internet application. In traditional storage technology, when users generate a lot of data, they will face the situation of insufficient local storage devices.

[0003] Chinese Patent Publication No.: CN106708428A discloses a data virtualization storage method and device, the method comprising: obtaining a storage request sent by a terminal; obtaining the data to be stored according to the storage request, and extracting the virtual storage path and virtual storage instruction in the storage request; searching for a distributed storage path corresponding to the virtual storage path, and searching for a distributed storage instruction corresponding to the virtual storage instruction in a distributed instruction library; storing the data to be stored in a distributed storage device according to the distributed storage path and the distributed storage instruction. The data virtualization storage method and device provided by the present invention can connect a terminal with a distributed storage device, and provide a distributed storage function for the terminal, and store the storage data sent by the terminal in the distributed storage device; it can be seen that the data virtualization storage method and device have the following problems: no memory planning is performed for the distributed storage device, which affects the IO performance of the parallel data warehouse, and further affects the running speed of the parallel distributed data warehouse. Summary of the invention

[0004] To this end, the present invention provides a method for improving the IO performance of a parallel data warehouse using distributed virtual storage, so as to overcome the problem in the prior art that no memory planning is performed for distributed storage devices, which affects the IO performance of the parallel data warehouse and further affects the running speed of the parallel distributed data warehouse.

[0005] To achieve the above object, the present invention provides a method for improving the IO performance of a parallel data warehouse by using distributed virtual storage, comprising:

[0006] Step S1, the central control module calculates the database IO performance evaluation parameter P according to the number of bytes read in a single database and the average transmission rate, and determines whether the database operation status meets the preset standard according to P;

[0007] Step S2, the central control module adjusts the warning level to the corresponding alarm level according to the calculated proportion of the total interruption duration of several transmission interruption processes occurring during the data transmission process of the database to the preset duration, and adjusts the preset evaluation parameters of the database according to the number of bytes of data read from the database at a single time;

[0008] Step S3, the central control module adjusts the overflow data of the database to the subsidiary database of the database according to the database IO performance evaluation parameter P, and adjusts the proportion of the reserved memory of a single database to the total memory of the server according to the total amount of the newly added subsidiary databases and the average evaluation parameter value of several databases in a single server;

[0009] Step S4, the central control module adjusts the data in the server according to the re-acquired average evaluation parameter value of a single server, the highest average evaluation parameter value among several servers, and the lowest average evaluation parameter value after the specific gravity is adjusted.

[0010] Furthermore, in step S1, the central control module calculates the database IO performance evaluation parameter P according to the number of bytes read by a single database at a time and the average transmission rate. The calculation formula of P is: Among them, D i is the average transmission rate of the database in the i-th preset period, set i = 1, 2, 3 ... n, Z is the number of bytes read by a single database at a time, α is the first evaluation correction coefficient, set α = 0.05s, β is the second evaluation correction coefficient, set β = 50kb -1 .

[0011] Furthermore, the central control module determines whether the database operation status meets the preset standard according to the database IO performance evaluation parameter P, wherein:

[0012] The first determination method is that the central control module determines that the operation status of the database meets the preset standard, and controls the database to operate with the current operation parameters;

[0013] The second determination method is that the central control module determines that the network environment of the server corresponding to the database does not meet the preset standard, calls the total interruption duration of several transmission interruption processes of the database during the data transmission process to calculate the proportion of the total interruption duration to the preset duration, and makes a secondary determination on whether the network environment of the server meets the preset standard according to the calculation result;

[0014] The third determination method is that the central control module determines that the operation state of the database does not meet the preset standard, the central control module obtains the total amount of data in the database, and adjusts the total amount of data in the database to a corresponding value according to the difference between the evaluation parameter P and the preset standard evaluation parameter;

[0015] The fourth determination method is that the central control module determines that the IO basic performance of the database does not meet the preset standard, and the central control module issues an alarm message for replacing the corresponding hardware and reduces the preset evaluation parameter to a corresponding value;

[0016] The first determination method satisfies that the database IO performance evaluation parameter P is greater than a third preset evaluation parameter;

[0017] The second determination method satisfies that the database IO performance evaluation parameter P is greater than the second preset evaluation parameter and is less than or equal to the third preset evaluation parameter;

[0018] The third determination method satisfies that the database IO performance evaluation parameter P is greater than the first preset evaluation parameter and is less than or equal to the second preset evaluation parameter;

[0019] The fourth determination method satisfies that the database IO performance evaluation parameter P is less than or equal to the first preset evaluation parameter;

[0020] Among them, the third preset evaluation parameter is greater than the second preset evaluation parameter which is greater than the first preset evaluation parameter.

[0021] Furthermore, the central control module determines the adjustment method for the warning level according to the proportion of the total interruption duration of several transmission interruption processes in the data transmission process of the acquired database to the preset duration under the second determination method, wherein:

[0022] The first adjustment method is that the central control module determines for the second time that the network environment of the server does not meet the preset standard, and changes the network working frequency band of the server corresponding to the database;

[0023] The second adjustment method is that the central control module determines that the home appliance interferes with the network environment, adjusts the network risk warning level to the first warning level and issues a first-level warning notification for network fluctuation risk;

[0024] The third adjustment method is that the central control module determines that the antenna transmission power of the existing network supply end does not meet the preset requirements, adjusts the network risk warning level to the second warning level and issues a second-level warning notification for network fluctuation risk;

[0025] The first adjustment method satisfies that the proportion of the total interruption duration to the preset duration is less than or equal to the first preset duration proportion;

[0026] The second adjustment method satisfies that the proportion of the total interruption duration to the preset duration is less than or equal to the second preset duration proportion and greater than the first preset duration proportion;

[0027] The third adjustment method satisfies that the proportion of the total interruption duration to the preset duration is greater than the proportion of the second preset duration;

[0028] Among them, the first preset duration ratio is smaller than the second preset duration ratio.

[0029] Furthermore, the central control module determines the adjustment method of the total data volume in the database according to the difference between the evaluation parameter P and the preset standard evaluation parameter in the third determination method, wherein:

[0030] The first data volume adjustment method is that the central control module adjusts the total data volume in the database to a preset total data volume, and stores the overflow data in a subsidiary database of the newly created database;

[0031] The second data volume adjustment method is that the central control module uses the first adjustment coefficient to adjust the total data volume in the database to the corresponding value, and stores the overflow data in the subsidiary database of the newly created database;

[0032] The third data volume adjustment method is that the central control module uses the second adjustment coefficient to adjust the total data volume in the database to the corresponding value, and stores the overflow data in the subsidiary database of the newly created database;

[0033] The first data volume adjustment method satisfies that the difference between the evaluation parameter P and the preset standard evaluation parameter is less than or equal to the first preset difference;

[0034] The second data volume adjustment method satisfies that the difference between the evaluation parameter P and the preset standard evaluation parameter is less than or equal to the second preset difference, and greater than the first preset difference;

[0035] The third data volume adjustment method satisfies that the difference between the evaluation parameter P and the preset standard evaluation parameter is greater than the second preset difference;

[0036] Among them, the first preset difference is smaller than the second preset difference, and the first adjustment coefficient is smaller than the second adjustment coefficient.

[0037] Furthermore, the central control module determines the adjustment mode of the server according to the total amount of newly added subsidiary databases in the acquired single server, wherein:

[0038] The first server adjustment mode is that the central control module determines that the operation state of the server meets the preset standard, and the central control module controls the server to operate with the current operation parameters;

[0039] The second server adjustment mode is that the central control module determines that the operation state of the server does not meet the preset standard, and the central control module adjusts the proportion of the reserved memory of a single database to the total memory of the server to a corresponding value according to the obtained evaluation parameter P of the single database and the average evaluation parameter value of several databases in the server;

[0040] The first server adjustment method satisfies that the total amount of newly added subsidiary databases in the server is less than or equal to a preset standard amount;

[0041] The second server adjustment method satisfies the requirement that the total amount of newly added subsidiary databases in the server is greater than a preset standard amount.

[0042] Furthermore, the central control module uses the proportion adjustment coefficient Q to calculate the proportion of the reserved memory of a single database to the total memory of the server in the second server adjustment mode, wherein the calculation formula of Q is: in, is the average evaluation parameter value of several databases in a single server, and γ is set to 10;

[0043] The central control module uses the proportion adjustment coefficient to adjust the proportion of the reserved memory of each single database in a single server to the total memory of the server to a corresponding value.

[0044] Furthermore, the central control module determines a secondary determination method for the server operation status according to the re-acquired average evaluation parameter values ​​of several databases in a single server in the second server adjustment mode, wherein:

[0045] The second determination method of the first state is that the central control module determines that the operation state of the server meets the preset standard, and the central control module controls the server to operate with the current operation parameters;

[0046] The second state secondary determination method is that the central control module determines that the operation state of the server does not meet the preset standard, and determines the adjustment method for the server data according to the highest average evaluation parameter value and the lowest average evaluation parameter value among the obtained servers;

[0047] The third state secondary determination method is that the central control module determines that the network environment of the server does not meet the preset standard;

[0048] The second determination method of the first state satisfies that the average evaluation of the database in the server is greater than the second preset average evaluation;

[0049] The second state secondary determination method satisfies that the average evaluation of the database in the server is greater than the first preset average evaluation and is less than or equal to the second preset evaluation;

[0050] The third state secondary determination method satisfies that the average evaluation of the database in the server is less than or equal to the first preset average evaluation;

[0051] Among them, the first preset average evaluation is less than the second preset average evaluation.

[0052] Furthermore, the central control module arranges the servers in descending order according to the average evaluation parameter values ​​of several databases in a single server obtained in sequence in the second state secondary determination mode to obtain the highest average evaluation parameter value. and the lowest average evaluation parameter value And determine the adjustment method for the server data according to the evaluation difference between the highest average evaluation parameter value and the lowest average evaluation parameter value, where:

[0053] The first server data adjustment method is that the central control module uses the first adjustment coefficient to coordinate the data of the server corresponding to the lowest average evaluation parameter value obtained by the central control module to the newly created subsidiary server;

[0054] The second server data adjustment method is that the central control module uses the second adjustment coefficient to adjust the data of the server corresponding to the highest average evaluation parameter value obtained by the central control module to the server corresponding to the lowest average evaluation parameter value;

[0055] The third server data adjustment method is that the central control module uses the third adjustment coefficient to adjust the data of the server corresponding to the lowest average evaluation parameter value obtained by the central control module to the server corresponding to the highest average evaluation parameter value;

[0056] The first server data adjustment method satisfies that the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value is less than or equal to a first preset evaluation difference value;

[0057] The second server data adjustment method satisfies that the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value is less than or equal to the second preset evaluation difference and greater than the first preset evaluation difference;

[0058] The third server data adjustment method satisfies that the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value is greater than the second preset evaluation difference value;

[0059] Among them, the first preset evaluation difference is smaller than the second preset evaluation difference; the second adjustment coefficient is smaller than the third adjustment coefficient which is smaller than the first adjustment coefficient.

[0060] Further, the central control module obtains the number of bytes read from the database at a single time under the fourth determination mode, and determines the adjustment mode for the preset evaluation parameter according to the difference between the preset standard number of bytes and the number of bytes read from the database at a single time, wherein:

[0061] The first adjustment method is that the central control module uses the first preset evaluation parameter adjustment coefficient to adjust the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values ​​respectively;

[0062] The second adjustment method is that the central control module uses the second preset evaluation parameter adjustment coefficient to adjust the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values ​​respectively;

[0063] The third adjustment mode is that the central control module adjusts the first preset evaluation parameter to the first corresponding evaluation parameter, the second preset evaluation parameter to the second corresponding evaluation parameter, and the third preset evaluation parameter to the third corresponding evaluation parameter;

[0064] The first adjustment method satisfies that the difference between the preset standard number of bytes and the number of bytes read from the database at a single time is less than or equal to a first preset byte difference;

[0065] The second adjustment method satisfies that the difference between the preset standard number of bytes and the number of bytes read from the database at a single time is less than or equal to the second preset byte difference, and is greater than the first preset byte difference;

[0066] The third adjustment method satisfies that the difference between the preset standard number of bytes and the number of bytes read from the database at a single time is greater than the second preset byte difference;

[0067] Among them, the first preset byte difference is smaller than the second preset byte difference, and the first preset evaluation adjustment coefficient is larger than the second preset evaluation adjustment coefficient.

[0068] Compared with the prior art, the beneficial effects of the present invention are that the central control module calculates the database IO performance evaluation parameter P according to the number of bytes read at a single time of a single database and the average transmission rate, and determines whether the database operation state meets the preset standard according to P, the central control module adjusts the alarm level to the corresponding alarm level according to the proportion of the total interruption duration of several transmission interruption processes in the data transmission process of the database to the preset duration, and adjusts the preset evaluation parameters of the database according to the number of bytes of data read at a single time by the database; the central control module adjusts the overflow data of the database to the subsidiary database of the database according to the database IO performance evaluation parameter P, and adjusts the proportion of the reserved memory of a single database to the total memory of the server according to the total amount of newly added subsidiary databases and the average evaluation parameter value of several databases in a single server; the central control module adjusts the data in the server according to the average evaluation parameter value of a single server re-acquired after the relative gravity adjustment, the highest average evaluation parameter value among several servers, and the lowest average evaluation parameter value, while effectively improving the IO performance of the parallel data warehouse, further improving the operation speed of the parallel distributed data warehouse, and planning and processing the electronic digital data to achieve rational resource allocation of servers and databases.

[0069] Furthermore, the central control module calculates the database IO performance evaluation parameter P according to the number of bytes read at a single time of a single database and the average transmission rate, thereby ensuring that the IO performance of a single database during detection can be accurately obtained and further improving the operating speed of the parallel distributed data warehouse.

[0070] Furthermore, the central control module determines whether the database operation status meets the preset standard based on P. When P is greater than a third preset evaluation parameter, the database operation status meets the preset standard; when P is less than or equal to the third preset evaluation parameter and greater than the second preset evaluation parameter, the central control module determines that the database network environment does not meet the preset standard; if P is less than or equal to the second preset evaluation parameter and greater than the first preset evaluation parameter, the central control module determines that the single database is overloaded; if the evaluation parameter P is less than or equal to the first preset evaluation parameter, the central control module determines that the database IO basic performance does not meet the preset standard; while ensuring that the cause of the poor database operation status is accurately obtained, the operation speed of the parallel distributed data warehouse is further improved.

[0071] Furthermore, when the central control module determines that the network environment of the server does not meet the preset standards, it determines the adjustment method for the alarm level based on the proportion of the total interruption duration of several transmission interruption processes in the data transmission process of the acquired database to the preset duration. While ensuring that the server works in a stable network state and promptly discovers the cause of network fluctuations, it further improves the operation speed of the parallel distributed data warehouse.

[0072] Furthermore, when the central control module determines that P is less than or equal to the second preset evaluation parameter and greater than the first preset evaluation parameter, the total data volume of the database is adjusted to a corresponding value according to the difference between P and the preset standard evaluation parameter, and the data is arranged in descending order according to the storage time to transfer the prematurely stored data that exceeds the adjusted total data volume to a newly created subsidiary database corresponding to the database; while ensuring that the database does not reduce IO performance due to excessive storage of data, the operating speed of the parallel distributed data warehouse is further improved.

[0073] Furthermore, the central control module determines the server adjustment method according to the total amount of newly added subsidiary databases in a single server, effectively avoiding the reduction of data transmission rate due to too many newly added databases and excessive server load, and further improving the operation speed of the parallel distributed data warehouse.

[0074] Furthermore, the central control module uses the proportion adjustment coefficient Q to calculate the proportion of a single database reserved memory to the total server memory under the second server adjustment mode, thereby further improving the operating speed of the parallel distributed data warehouse while ensuring reasonable memory allocation.

[0075] Furthermore, after the central control module adjusts the proportion of the reserved memory of each individual database in a single server to the total memory of the server to a corresponding value, the central control module determines a secondary judgment method for the server operation status according to the re-acquired average evaluation parameter value of several databases in the single server, thereby further improving the operation speed of the parallel distributed data warehouse while ensuring reasonable data planning.

[0076] Furthermore, the central control module, in the second server secondary determination mode, obtains the highest average evaluation parameter value. and the lowest average evaluation parameter value The difference in the value determines the adjustment method for the server data, which further rationalizes the memory allocation and further improves the IO performance of the parallel data warehouse.

[0077] Furthermore, when the central control module determines that the IO basic performance of the server corresponding to the database is too low, the preset evaluation parameter is reduced to a corresponding value based on the difference between the preset standard number of bytes and the number of bytes read by the database in a single time, thereby ensuring that the method can be applied to a wider range of areas while further improving the operating speed of the parallel distributed data warehouse. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A flowchart of the steps of a method for improving the IO performance of a parallel data warehouse using distributed virtual storage according to an embodiment of the present invention;

[0079] Figure 2 A flow chart of a method for determining a database operation status in a method for improving IO performance of a parallel data warehouse using distributed virtual storage according to an embodiment of the present invention;

[0080] Figure 3 A flow chart of a method for adjusting the total amount of data in a database in a method for improving the IO performance of a parallel data warehouse using distributed virtual storage according to an embodiment of the present invention;

[0081] Figure 4 This is a flow chart of a state secondary determination method of a method for improving the IO performance of a parallel data warehouse using distributed virtual storage according to an embodiment of the present invention. DETAILED DESCRIPTION

[0082] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0083] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0084] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0085] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0086] See also Figure 1 , Figure 2 , Figure 3 as well as Figure 4 As shown, they are respectively a flowchart of the steps of a method for improving the IO performance of a parallel data warehouse using distributed virtual storage according to an embodiment of the present invention, a flowchart of a method for determining the operation status of a database, a flowchart of a method for adjusting the total amount of data in the database, and a flowchart of a method for secondary determination of the status; a method for improving the IO performance of a parallel data warehouse using distributed virtual storage according to an embodiment of the present invention, comprising:

[0087] Step S1, the central control module calculates the database IO performance evaluation parameter P according to the number of bytes read by a single database and the average transmission rate, and determines whether the database operation status meets the preset standard according to P. The parallel distributed data warehouse includes several servers, and a single server contains several databases, which are used to store data in a preset storage path;

[0088] Step S2, the central control module adjusts the warning level to the corresponding alarm level according to the calculated proportion of the total interruption duration of several transmission interruption processes occurring during the data transmission process of the database to the preset duration, and adjusts the preset evaluation parameters of the database according to the number of bytes of data read from the database at a single time;

[0089] Step S3, the central control module adjusts the overflow data of the database to the subsidiary database of the database according to the database IO performance evaluation parameter P, and adjusts the proportion of the reserved memory of a single database to the total memory of the server according to the total amount of the newly added subsidiary databases and the average evaluation parameter value of several databases in a single server;

[0090] Step S4, the central control module adjusts the data in the server according to the re-acquired average evaluation parameter value of a single server, the highest average evaluation parameter value among several servers, and the lowest average evaluation parameter value after the specific gravity is adjusted.

[0091] Specifically, in step S1, the central control module calculates the database IO performance evaluation parameter P according to the number of bytes read by a single database at a time and the average transmission rate. The calculation formula of P is: Among them, D i is the average transmission rate of the database in the i-th preset period, set i = 1, 2, 3 ... n, Z is the number of bytes read by a single database at a time, α is the first evaluation correction coefficient, set α = 0.05s, β is the second evaluation correction coefficient, set β = 50kb -1 .

[0092] Specifically, the central control module determines whether the database operation status meets the preset standard according to the database IO performance evaluation parameter P, wherein:

[0093] The first determination method is that the central control module determines that the operation status of the database meets the preset standard, and controls the database to operate with the current operation parameters;

[0094] The second determination method is that the central control module determines that the network environment of the server corresponding to the database does not meet the preset standard, calls the total interruption duration of several transmission interruption processes of the database during the data transmission process to calculate the proportion of the total interruption duration to the preset duration, and makes a secondary determination on whether the network environment of the server meets the preset standard according to the calculation result;

[0095] The third determination method is that the central control module determines that the operation state of the database does not meet the preset standard, the central control module obtains the total amount of data in the database, and adjusts the total amount of data in the database to a corresponding value according to the difference between the evaluation parameter P and the preset standard evaluation parameter;

[0096] The fourth determination method is that the central control module determines that the IO basic performance of the database does not meet the preset standard, and the central control module issues an alarm message for replacing the corresponding hardware and reduces the preset evaluation parameter to a corresponding value;

[0097] The first determination method satisfies that the database IO performance evaluation parameter P is greater than a third preset evaluation parameter;

[0098] The second determination method satisfies that the database IO performance evaluation parameter P is greater than the second preset evaluation parameter and is less than or equal to the third preset evaluation parameter;

[0099] The third determination method satisfies that the database IO performance evaluation parameter P is greater than the first preset evaluation parameter and is less than or equal to the second preset evaluation parameter;

[0100] The fourth determination method satisfies that the database IO performance evaluation parameter P is less than or equal to the first preset evaluation parameter;

[0101] Among them, the third preset evaluation parameter is greater than the second preset evaluation parameter which is greater than the first preset evaluation parameter.

[0102] Specifically, the central control module determines the adjustment method for the warning level according to the proportion of the total interruption duration of several transmission interruption processes in the data transmission process of the acquired database to the preset duration under the second determination method, wherein:

[0103] The first adjustment method is that the central control module determines for the second time that the network environment of the server does not meet the preset standard, and changes the network working frequency band of the server corresponding to the database;

[0104] The second adjustment method is that the central control module determines that the home appliance interferes with the network environment, adjusts the network risk warning level to the first warning level and issues a first-level warning notification for network fluctuation risk;

[0105] The third adjustment method is that the central control module determines that the antenna transmission power of the existing network supply end does not meet the preset requirements, adjusts the network risk warning level to the second warning level and issues a second-level warning notification for network fluctuation risk;

[0106] The first adjustment method satisfies that the proportion of the total interruption duration to the preset duration is less than or equal to the first preset duration proportion;

[0107] The second adjustment method satisfies that the proportion of the total interruption duration to the preset duration is less than or equal to the second preset duration proportion and greater than the first preset duration proportion;

[0108] The third adjustment method satisfies that the proportion of the total interruption duration to the preset duration is greater than the proportion of the second preset duration;

[0109] Among them, the first preset duration ratio is smaller than the second preset duration ratio.

[0110] Specifically, the central control module determines the adjustment method of the total data volume in the database according to the difference between the evaluation parameter P and the preset standard evaluation parameter in the third determination method, wherein:

[0111] The first data volume adjustment method is that the central control module adjusts the total data volume in the database to a preset total data volume, and stores the overflow data in a subsidiary database of the newly created database;

[0112] The second data volume adjustment method is that the central control module uses the first adjustment coefficient to adjust the total data volume in the database to the corresponding value, and stores the overflow data in the subsidiary database of the newly created database;

[0113] The third data volume adjustment method is that the central control module uses the second adjustment coefficient to adjust the total data volume in the database to the corresponding value, and stores the overflow data in the subsidiary database of the newly created database;

[0114] The first data volume adjustment method satisfies that the difference between the evaluation parameter P and the preset standard evaluation parameter is less than or equal to the first preset difference;

[0115] The second data volume adjustment method satisfies that the difference between the evaluation parameter P and the preset standard evaluation parameter is less than or equal to the second preset difference, and greater than the first preset difference;

[0116] The third data volume adjustment method satisfies that the difference between the evaluation parameter P and the preset standard evaluation parameter is greater than the second preset difference;

[0117] Among them, the first preset difference is smaller than the second preset difference, and the first adjustment coefficient is smaller than the second adjustment coefficient.

[0118] Specifically, the central control module determines the adjustment method of the server according to the total amount of newly added subsidiary databases in the acquired single server, wherein:

[0119] The first server adjustment mode is that the central control module determines that the operation state of the server meets the preset standard, and the central control module controls the server to operate with the current operation parameters;

[0120] The second server adjustment mode is that the central control module determines that the operation state of the server does not meet the preset standard, and the central control module adjusts the proportion of the reserved memory of a single database to the total memory of the server to a corresponding value according to the obtained evaluation parameter P of the single database and the average evaluation parameter value of several databases in the server;

[0121] The first server adjustment method satisfies that the total amount of newly added subsidiary databases in the server is less than or equal to a preset standard amount;

[0122] The second server adjustment method satisfies the requirement that the total amount of newly added subsidiary databases in the server is greater than a preset standard amount.

[0123] Specifically, the central control module uses the proportion adjustment coefficient Q to calculate the proportion of the reserved memory of a single database to the total memory of the server under the second server adjustment mode, wherein the calculation formula of Q is: in, is the average evaluation parameter value of several databases in a single server, and γ is set to 10;

[0124] The central control module uses the proportion adjustment coefficient to adjust the proportion of the reserved memory of each single database in a single server to the total memory of the server to a corresponding value.

[0125] Specifically, the central control module determines a secondary determination method for the server operation status according to the re-acquired average evaluation parameter values ​​of several databases in a single server under the second server adjustment mode, wherein:

[0126] The second determination method of the first state is that the central control module determines that the operation state of the server meets the preset standard, and the central control module controls the server to operate with the current operation parameters;

[0127] The second state secondary determination method is that the central control module determines that the operation state of the server does not meet the preset standard, and determines the adjustment method for the server data according to the highest average evaluation parameter value and the lowest average evaluation parameter value among the obtained servers;

[0128] The third state secondary determination method is that the central control module determines that the network environment of the server does not meet the preset standard;

[0129] The second determination method of the first state satisfies that the average evaluation of the database in the server is greater than the second preset average evaluation;

[0130] The second state secondary determination method satisfies that the average evaluation of the database in the server is greater than the first preset average evaluation and is less than or equal to the second preset evaluation;

[0131] The third state secondary determination method satisfies that the average evaluation of the database in the server is less than or equal to the first preset average evaluation;

[0132] Among them, the first preset average evaluation is less than the second preset average evaluation.

[0133] Specifically, the central control module arranges the servers in descending order according to the average evaluation parameter values ​​of several databases in a single server obtained in sequence in the second state secondary determination mode to obtain the highest average evaluation parameter value. and the lowest average evaluation parameter value And determine the adjustment method for the server data according to the evaluation difference between the highest average evaluation parameter value and the lowest average evaluation parameter value, where:

[0134] The first server data adjustment method is that the central control module uses the first adjustment coefficient to coordinate the data of the server corresponding to the lowest average evaluation parameter value obtained by the central control module to the newly created subsidiary server;

[0135] The second server data adjustment method is that the central control module uses the second adjustment coefficient to adjust the data of the server corresponding to the highest average evaluation parameter value obtained by the central control module to the server corresponding to the lowest average evaluation parameter value;

[0136] The third server data adjustment method is that the central control module uses the third adjustment coefficient to adjust the data of the server corresponding to the lowest average evaluation parameter value obtained by the central control module to the server corresponding to the highest average evaluation parameter value;

[0137] The first server data adjustment method satisfies that the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value is less than or equal to a first preset evaluation difference value;

[0138] The second server data adjustment method satisfies that the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value is less than or equal to the second preset evaluation difference and greater than the first preset evaluation difference;

[0139] The third server data adjustment method satisfies that the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value is greater than the second preset evaluation difference value;

[0140] Among them, the first preset evaluation difference is smaller than the second preset evaluation difference, and the second adjustment coefficient is smaller than the third adjustment coefficient which is smaller than the first adjustment coefficient.

[0141] Specifically, the central control module obtains the number of bytes read from the database at a single time under the fourth determination method, and determines the adjustment method for the preset evaluation parameter according to the difference between the preset standard number of bytes and the number of bytes read from the database at a single time, wherein:

[0142] The first adjustment method is that the central control module uses the first preset evaluation parameter adjustment coefficient to adjust the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values ​​respectively;

[0143] The second adjustment method is that the central control module uses the second preset evaluation parameter adjustment coefficient to adjust the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values ​​respectively;

[0144] The third adjustment mode is that the central control module adjusts the first preset evaluation parameter to the first corresponding evaluation parameter, the second preset evaluation parameter to the second corresponding evaluation parameter, and the third preset evaluation parameter to the third corresponding evaluation parameter;

[0145] The first adjustment method satisfies that the difference between the preset standard number of bytes and the number of bytes read from the database at a single time is less than or equal to a first preset byte difference;

[0146] The second adjustment method satisfies that the difference between the preset standard number of bytes and the number of bytes read from the database at a single time is less than or equal to the second preset byte difference, and is greater than the first preset byte difference;

[0147] The third adjustment method satisfies that the difference between the preset standard number of bytes and the number of bytes read from the database at a single time is greater than the second preset byte difference;

[0148] Among them, the first preset byte difference is less than the second preset byte difference, and the first preset evaluation adjustment coefficient is greater than the second preset evaluation adjustment coefficient. So far, the technical scheme of the present invention has been described in conjunction with the preferred implementation modes shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific implementation modes. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical schemes after these changes or substitutions will fall within the protection scope of the present invention.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for improving the IO performance of a parallel data warehouse using distributed virtual storage, characterized in that: include: Step S1, the central control module calculates the database IO performance evaluation parameter P according to the number of bytes read by a single database at a time and the average transmission rate, and determines whether the database operation status meets the preset standard according to P; Step S2, the central control module adjusts the warning level to the corresponding alarm level according to the calculated proportion of the total interruption duration of several transmission interruption processes occurring during the data transmission process of the database to the preset duration, and adjusts the preset evaluation parameters of the database according to the number of bytes of data read from the database at a single time; Step S3, the central control module adjusts the overflow data of the database to the subsidiary database of the database according to the database IO performance evaluation parameter P, and adjusts the proportion of the reserved memory of a single database to the total memory of the server according to the total amount of the newly added subsidiary databases and the average evaluation parameter value of several databases in a single server; Step S4, the central control module adjusts the data in the server according to the re-acquired average evaluation parameter value of a single server, the highest average evaluation parameter value among several servers, and the lowest average evaluation parameter value after adjusting the specific gravity; The central control module determines whether the database operation status meets the preset standard according to the database IO performance evaluation parameter P, wherein: The first determination method is that the central control module determines that the operation status of the database meets the preset standard, and controls the database to operate with the current operation parameters; The second determination method is that the central control module determines that the network environment of the server corresponding to the database does not meet the preset standard, calls the total interruption duration of several transmission interruption processes of the database during the data transmission process to calculate the proportion of the total interruption duration to the preset duration, and makes a secondary determination on whether the network environment of the server meets the preset standard according to the calculation result; The third determination method is that the central control module determines that the operation state of the database does not meet the preset standard, the central control module obtains the total amount of data in the database, and adjusts the total amount of data in the database to a corresponding value according to the difference between the evaluation parameter P and the preset standard evaluation parameter; The fourth determination method is that the central control module determines that the IO basic performance of the database does not meet the preset standard, and the central control module issues an alarm message for replacing the corresponding hardware and reduces the preset evaluation parameter to a corresponding value; The first determination method satisfies that the database IO performance evaluation parameter P is greater than a third preset evaluation parameter; The second determination method satisfies that the database IO performance evaluation parameter P is greater than the second preset evaluation parameter and is less than or equal to the third preset evaluation parameter; The third determination method satisfies that the database IO performance evaluation parameter P is greater than the first preset evaluation parameter and is less than or equal to the second preset evaluation parameter; The fourth determination method satisfies that the database IO performance evaluation parameter P is less than or equal to the first preset evaluation parameter; Wherein, the third preset evaluation parameter is greater than the second preset evaluation parameter and greater than the first preset evaluation parameter; The central control module determines the adjustment method for the warning level according to the proportion of the total interruption duration of several transmission interruption processes in the data transmission process of the acquired database to the preset duration in the second determination method, wherein: The first adjustment method is that the central control module determines for the second time that the network environment of the server does not meet the preset standard, and changes the network working frequency band of the server corresponding to the database; The second adjustment method is that the central control module determines that the home appliance interferes with the network environment, adjusts the network risk warning level to the first warning level and issues a first-level warning notification for network fluctuation risk; The third adjustment method is that the central control module determines that the antenna transmission power of the existing network supply end does not meet the preset requirements, adjusts the network risk warning level to the second warning level and issues a second-level warning notification for network fluctuation risk; The first adjustment method satisfies that the proportion of the total interruption duration to the preset duration is less than or equal to the first preset duration proportion; The second adjustment method satisfies that the proportion of the total interruption duration to the preset duration is less than or equal to the second preset duration proportion and greater than the first preset duration proportion; The third adjustment method satisfies that the proportion of the total interruption duration to the preset duration is greater than the proportion of the second preset duration; Wherein, the first preset duration ratio is less than the second preset duration ratio; The central control module determines the adjustment method of the total data volume in the database according to the difference between the evaluation parameter P and the preset standard evaluation parameter in the third determination method, wherein: The first data volume adjustment method is that the central control module adjusts the total data volume in the database to a preset total data volume, and stores the overflow data in a subsidiary database of the newly created database; The second data volume adjustment method is that the central control module uses the first adjustment coefficient to adjust the total data volume in the database to the corresponding value, and stores the overflow data in the subsidiary database of the newly created database; The third data volume adjustment method is that the central control module uses the second adjustment coefficient to adjust the total data volume in the database to the corresponding value, and stores the overflow data in the subsidiary database of the newly created database; The first data volume adjustment method satisfies that the difference between the evaluation parameter P and the preset standard evaluation parameter is less than or equal to the first preset difference; The second data volume adjustment method satisfies that the difference between the evaluation parameter P and the preset standard evaluation parameter is less than or equal to the second preset difference, and greater than the first preset difference; The third data volume adjustment method satisfies that the difference between the evaluation parameter P and the preset standard evaluation parameter is greater than the second preset difference; Wherein, the first preset difference is smaller than the second preset difference, and the first adjustment coefficient is smaller than the second adjustment coefficient; The central control module determines the adjustment method of the server according to the total amount of newly added subsidiary databases in the single server obtained, where: The first server adjustment mode is that the central control module determines that the operation state of the server meets the preset standard, and the central control module controls the server to operate with the current operation parameters; The second server adjustment mode is that the central control module determines that the operation state of the server does not meet the preset standard, and the central control module adjusts the proportion of the reserved memory of a single database to the total memory of the server to a corresponding value according to the obtained evaluation parameter P of the single database and the average evaluation parameter value of several databases in the server; The first server adjustment method satisfies that the total amount of newly added subsidiary databases in the server is less than or equal to a preset standard amount; The second server adjustment mode satisfies that the total amount of newly added subsidiary databases in the server is greater than a preset standard amount; The central control module uses the proportion adjustment coefficient Q to calculate the proportion of the reserved memory of a single database to the total memory of the server in the second server adjustment mode, wherein the calculation formula of Q is: in, is the average evaluation parameter value of several databases in a single server, and γ is set to 10; The central control module uses the proportion adjustment coefficient to adjust the proportion of the reserved memory of each single database in a single server to the total memory of the server to the corresponding value; The central control module determines a secondary determination method for the server operation status according to the re-acquired average evaluation parameter values ​​of several databases in a single server in the second server adjustment mode, wherein: The second determination method of the first state is that the central control module determines that the operation state of the server meets the preset standard, and the central control module controls the server to operate with the current operation parameters; The second state secondary determination method is that the central control module determines that the operation state of the server does not meet the preset standard, and determines the adjustment method for the server data according to the highest average evaluation parameter value and the lowest average evaluation parameter value among the obtained servers; The third state secondary determination method is that the central control module determines that the network environment of the server does not meet the preset standard; The second determination method of the first state satisfies that the average evaluation of the database in the server is greater than the second preset average evaluation; The second state secondary determination method satisfies that the average evaluation of the database in the server is greater than the first preset average evaluation and is less than or equal to the second preset evaluation; The third state secondary determination method satisfies that the average evaluation of the database in the server is less than or equal to the first preset average evaluation; Wherein, the first preset average evaluation is less than the second preset average evaluation; The central control module arranges the servers in descending order according to the average evaluation parameter values ​​of several databases in a single server obtained in sequence in the second state secondary determination mode to obtain the highest average evaluation parameter value. and the lowest average evaluation parameter value And determine the adjustment method for the server data according to the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value, where: The first server data adjustment method is that the central control module uses the first adjustment coefficient to coordinate the data of the server corresponding to the lowest average evaluation parameter value obtained by the central control module to the newly created subsidiary server; The second server data adjustment method is that the central control module uses the second adjustment coefficient to adjust the data of the server corresponding to the highest average evaluation parameter value obtained by the central control module to the server corresponding to the lowest average evaluation parameter value; The third server data adjustment method is that the central control module uses the third adjustment coefficient to adjust the data of the server corresponding to the lowest average evaluation parameter value obtained by the central control module to the server corresponding to the highest average evaluation parameter value; The first server data adjustment method satisfies that the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value is less than or equal to a first preset evaluation difference value; The second server data adjustment method satisfies that the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value is less than or equal to the second preset evaluation difference and greater than the first preset evaluation difference; The third server data adjustment method satisfies that the difference between the highest average evaluation parameter value and the lowest average evaluation parameter value is greater than the second preset evaluation difference value; Wherein, the first preset evaluation difference is smaller than the second preset evaluation difference, and the second adjustment coefficient is smaller than the third adjustment coefficient, which is smaller than the first adjustment coefficient; The central control module obtains the number of bytes read from the database at a single time under the fourth determination method, and determines the adjustment method for the preset evaluation parameter according to the difference between the preset standard number of bytes and the number of bytes read from the database at a single time, wherein: The first adjustment method is that the central control module uses the first preset evaluation parameter adjustment coefficient to adjust the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values ​​respectively; The second adjustment method is that the central control module uses the second preset evaluation parameter adjustment coefficient to adjust the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values ​​respectively; The third adjustment mode is that the central control module adjusts the first preset evaluation parameter to the first corresponding evaluation parameter, the second preset evaluation parameter to the second corresponding evaluation parameter, and the third preset evaluation parameter to the third corresponding evaluation parameter; The first adjustment method satisfies that the difference between the preset standard number of bytes and the number of bytes read from the database at a single time is less than or equal to a first preset byte difference; The second adjustment method satisfies that the difference between the preset standard number of bytes and the number of bytes read from the database at a single time is less than or equal to the second preset byte difference, and is greater than the first preset byte difference; The third adjustment method satisfies that the difference between the preset standard number of bytes and the number of bytes read from the database at a single time is greater than the second preset byte difference; Among them, the first preset byte difference is smaller than the second preset byte difference, and the first preset evaluation adjustment coefficient is larger than the second preset evaluation adjustment coefficient.

2. The method for improving parallel data warehouse IO performance using distributed virtual storage according to claim 1, characterized in that: In step S1, the central control module calculates the database IO performance evaluation parameter P according to the number of bytes read by a single database at a time and the average transmission rate. The calculation formula of P is: Among them, D i is the average transmission rate of the database in the i-th preset period, set i = 1, 2, 3 ... n, Z is the number of bytes read by a single database at a time, α is the first evaluation correction coefficient, set α = 0.05s, β is the second evaluation correction coefficient, set β = 50kb -1 .