Data volume evaluation method, system and equipment based on box line method

Abstract

The invention relates to the technical field of data management, and discloses a data volume evaluation method, system and equipment based on a box line method. The method comprises the steps of determining a to-be-tested data table; newly establishing a data table volume statistical information table and a data quality detection table; monitoring the to-be-monitored data table based on the data volume collection script, the data volume monitoring script and the data table volume statistical information table to obtain a monitoring result; the monitoring result comprises a volume reference value of the to-be-detected data table and a confidence interval of the t-day data volume of the to-be-detected data table calculated based on a box plot method; storing the process parameters and the monitoring result in a data quality detection table; and reading the data quality detection table based on the alarm service, and generating real-time alarm information. According to the method, the confidence interval of the data volume is calculated based on the box line method, and the data volume of the data table is automatically detected. Whether the data volume is normal or not is comprehensively judged through the line number and the data byte number, rough problem positioning is given, and implementation is easy.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a data volume assessment method, system and device based on a boxplot method. Background Art

[0002] In data analysis and data management, the correctness of data volume is crucial to ensure the accuracy and reliability of analysis results. However, the traditional method mainly relies on the experience and intuition of data analysts to judge whether the data volume is abnormal. This method is extremely dependent on the data analyst's familiarity with the data. When there are many categories of data tables, manual verification is not advisable, and there are problems such as low efficiency and large errors. Summary of the invention

[0003] The main purpose of the present invention is to provide a data volume assessment method, system and device based on the box-and-whisker method, aiming to solve at least one of the above problems.

[0004] To achieve the above object, the present invention provides a data volume assessment method based on the boxplot method, comprising:

[0005] Determine the data table to be tested;

[0006] Create a new data table volume statistics table and data quality detection table;

[0007] Based on the data volume collection script, the data volume monitoring script and the data table volume statistical information table, the data table to be tested is monitored to obtain a monitoring result; wherein the monitoring result includes a volume benchmark value of the data table to be tested and a confidence interval of the data volume of the data table to be tested on day t calculated based on the box-line method;

[0008] Saving the process parameters and the monitoring results to the data quality detection table;

[0009] The data quality detection table is read based on the alarm service to generate real-time alarm information.

[0010] In some embodiments, the newly created data table volume statistics information table and data quality detection table include:

[0011] Calling a service interface to read the number of data rows and bytes in a preset partition of the data table to be tested;

[0012] Use date, database instance name, database name, table name, and preset partition as conditions to query whether records already exist in the data table volume statistics table;

[0013] If the record already exists, delete the existing data and save the number of rows and bytes of the read data to the data table volume statistics table;

[0014] If no record exists, the number of rows and bytes of the read data is saved in the data table volume statistics table;

[0015] Create a new data quality detection table.

[0016] In some embodiments, the monitoring of the data table to be tested based on the data volume collection script, the data volume monitoring script and the data table volume statistical information table to obtain the monitoring result includes:

[0017] Write data volume collection scripts and data volume monitoring scripts;

[0018] Configuring the data volume collection script and the data volume monitoring script into the task flow scheduling system;

[0019] Run the data volume collection script and the data volume monitoring script, monitor the data table to be tested according to the data table volume statistical information table, and obtain process parameters and monitoring results.

[0020] In some embodiments, the running of the data volume collection script and the data volume monitoring script, monitoring the data table to be tested according to the data table volume statistical information table, and obtaining process parameters and monitoring results, includes:

[0021] Running the data volume collection script;

[0022] Based on the data volume collection script, the volume data of the preset partition of the data table to be tested on a preset date is collected, and the volume data is saved to the data table volume statistical information table;

[0023] Run the data volume monitoring script;

[0024] The data table to be tested is monitored based on the data volume monitoring script and the data table volume statistical information table to obtain process parameters and monitoring results.

[0025] In some embodiments, the step of collecting the volume data of a preset partition of the data table to be tested on a preset date based on the data volume collection script and saving the volume data to the data table volume statistics information table includes:

[0026] Read the records in the data table volume statistics information table based on the instance, database, table, and preset partition as conditions;

[0027] Selecting the record with the latest date from the records;

[0028] Calculate the volume reference value of the data table to be tested on day t based on the record of the latest date;

[0029] Calculate the confidence interval of the data volume on day t of the data table to be detected based on the records of the latest date based on the box plot method;

[0030] The volume benchmark value of the data table to be detected and the confidence interval of the data volume on day t of the data table to be detected are saved in the data table volume statistical information table.

[0031] In some embodiments, the volume reference value of the to-be-detected data table on day t is calculated:

[0032]

[0033] in, represents the estimated number of rows on day t; Δ represents the average number of bytes per row in the past 7 days; represents the estimated number of bytes on day t; S i Indicates the number of bytes; L i Indicates the number of rows; t-1 means the day before t, t-2 means 2 days before t, and t-7 means 7 days before t.

[0034] In some embodiments, the calculating the confidence interval of the data volume on day t of the to-be-detected data table according to the records of the latest date based on the box plot method includes:

[0035] Calculate the maximum value a_max, the upper quartile a3 / 4, the median a2 / 4, and the lower quartile a1 / 4 of the data bytes in the data table to be tested in the past 60 days, and calculate the harmonic upper limit A_max and the harmonic lower limit A_min based on the maximum value a_max, the upper quartile a3 / 4, the median a2 / 4, and the lower quartile a1 / 4 of the minimum value a_min:

[0036] A_max=max(a_max,a3 / 4+a3 / 4-a2 / 4)

[0037] A_min=min(a_min,a1 / 4-(a2 / 4-a1 / 4))

[0038] Calculate the confidence interval of the data volume on day t of the data table to be tested:

[0039] CI = (A_min, A_max)

[0040] Among them, CI is the confidence interval of the data volume on day t in the data table to be tested.

[0041] In some embodiments, the step of reading the data quality detection table based on the alarm service and generating real-time alarm information includes:

[0042] Read the data quality detection table based on the alarm service to obtain the confidence interval of the data volume on day t of the data table to be detected;

[0043] Compare the data volume of the day with the confidence interval;

[0044] If the data volume size of the day falls within the confidence interval, the data volume detection of the day passes;

[0045] If the data volume for the day is less than the harmonic lower limit of the confidence interval or greater than the harmonic upper limit of the confidence interval, a real-time alarm message is generated.

[0046] In addition, to achieve the above purpose, the present invention also proposes a data volume evaluation system based on the box-line method, comprising:

[0047] A determination module, used for determining a data table to be tested;

[0048] Create a new module to create a new data table volume statistics table and data quality detection table;

[0049] A monitoring module, used to monitor the data table to be tested based on a data volume collection script, a data volume monitoring script and the data table volume statistical information table to obtain a monitoring result; the monitoring result includes a volume benchmark value of the data table to be tested and a confidence interval of the data volume of the data table to be tested on day t calculated based on the box plot method;

[0050] A saving module, used for saving the monitoring process parameters and the monitoring results to the data quality detection table;

[0051] The alarm module is used to read the data quality detection table based on the alarm service and generate real-time alarm information.

[0052] In addition, to achieve the above-mentioned purpose, the present invention also proposes an electronic device, which includes: a memory, a processor, and a data volume assessment program based on the box-and-whisker method stored in the memory and executable on the processor, wherein the data volume assessment program based on the box-and-whisker method is configured to implement the data volume assessment method based on the box-and-whisker method as described above.

[0053] The present invention provides a data volume assessment method based on the box-line method, including: determining a data table to be tested; creating a data table volume statistical information table and a data quality detection table; monitoring the data table to be tested based on a data volume collection script, a data volume monitoring script and the data table volume statistical information table to obtain a monitoring result; wherein the monitoring result includes a volume baseline value of the data table to be tested and a confidence interval of the data volume of the data table to be tested on day t calculated based on the box-line method; saving process parameters and the monitoring results to the data quality detection table; reading the data quality detection table based on an alarm service to generate real-time alarm information. In the present invention, the statistical confidence interval of the data volume is calculated based on the box-line method, and combined with the growth model of the data volume of the data table, the data volume of the data table is automatically detected, and the detection efficiency is faster and the accuracy is higher. Compared with the traditional method, the present invention uses the number of rows and the number of data bytes to comprehensively determine whether the data volume is normal, and can also give a rough problem location. The method is simple and easy to understand and implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the structure of an electronic device in a hardware operating environment involved in an embodiment of the present invention;

[0055] Figure 2 It is a flow chart of an embodiment of a method for evaluating data volume based on the box-and-line method of the present invention;

[0056] Figure 3 It is a schematic diagram of the overall process involved in the embodiment of the present invention;

[0057] Figure 4 It is a schematic diagram of the S2 process in the overall process involved in the embodiment of the present invention;

[0058] Figure 5 It is a schematic diagram of the S3 process in the overall process involved in the embodiment of the present invention;

[0059] Figure 6 It is a structural block diagram of an embodiment of a data volume assessment system based on the box-and-whisker method of the present invention.

[0060] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0063] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0064] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of an electronic device of a hardware operating environment involved in an embodiment of the present invention.

[0065] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM memory) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0066] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0067] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a data volume assessment program based on the box-line method.

[0068] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device, and the electronic device calls the data volume assessment program based on the box-and-whisker method stored in the memory 1005 through the processor 1001, and executes the data volume assessment method based on the box-and-whisker method provided in an embodiment of the present invention.

[0069] The present invention proposes a data volume assessment method, system and device based on the boxplot method.

[0070] The present invention provides a method for evaluating data volume based on the box-line method. Figure 2 , Figure 2 Schematic diagram of a flow chart of an embodiment of a data volume assessment method based on the box-and-whisker method of the present invention.

[0071] like Figure 2 As shown, the data volume evaluation method based on the box-line method includes:

[0072] Step S100: determining a data table to be tested;

[0073] Step S200: Create a new data table volume statistics information table and a data quality detection table;

[0074] Step S300: monitoring the data table to be tested based on the data volume collection script, the data volume monitoring script and the data table volume statistical information table to obtain monitoring results; wherein the monitoring results include the volume benchmark value of the data table to be tested and the confidence interval of the data volume of the data table to be tested on day t calculated based on the box-line method;

[0075] Step S400: saving the process parameters and the monitoring results to the data quality detection table;

[0076] Step S500: Read the data quality detection table based on the alarm service to generate real-time alarm information.

[0077] It should be noted that the execution subject in this embodiment may be an electronic device, which may be a computer device with data processing functions, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment, a computer device is used as an example for explanation.

[0078] It is understandable that this embodiment uses the data volume assessment method based on the box-line method as an example to illustrate the case where the data table is partitioned by day (or month). Figure 3 As shown, the method includes: S1, determining the table T to be monitored for data volume; S2, creating a new table Q for data volume statistics; S3, writing a data volume collection script A; S4, writing a data volume monitoring script B; S5, configuring script A and script B to the task flow scheduling system; S6, the alarm service reads the data in the W table and issues a real-time alarm. By calculating the statistical confidence interval of the data volume and combining the growth model of the data table data volume, the data volume of the data table is automatically detected.

[0079] In one embodiment, a data table to be tested is determined.

[0080] Specifically, Figure 3 As shown, S1: determine the table T for which data volume monitoring is to be performed, and the ETL task corresponding to table T (data table to be tested) in the task flow scheduling system (such as Apache DolphinScheduler) is Task-T.

[0081] It should be noted that a data table is a structure used to store and organize data in a database management system (DBMS). In database design, data tables are the basis for building a database. They store data in the form of rows and columns, where each column represents a field, which defines the type of data (such as integers, strings, dates, etc.), and each row represents a record, which contains specific data of a specific entity.

[0082] In one embodiment, a new data table volume statistics information table and a data quality detection table are created, including: calling a service interface to read the number of data rows and bytes of the data table to be tested in a preset partition; using date, database instance name, database name, table name, and preset partition as conditions to query whether there are already records in the data table volume statistics information table; if there are already records, deleting the existing data and saving the read number of data rows and bytes to the data table volume statistics information table; if there are no records, saving the read number of data rows and bytes to the data table volume statistics information table; and creating a new data quality detection table.

[0083] Specifically, Figure 3As shown, S2: Create a data table volume statistics information table Q. The Q table shall at least include the following fields <instance (database instance name), db (database name), table (table name), pt (partition), len (number of rows), size (storage size in bytes), day (recording date)>; Create a data quality detection table W. The W table shall at least include the following fields <instance (database instance name), db (database name), table (table name), pt (partition), q_type (quality type), result (result), day (recording date)>.

[0084] Exemplarily, collect the statistical information of the data table volume from the data table T to be detected into the Q table (data table volume statistics information table). The Q table includes but is not limited to the following fields <instance (database instance name), db (database name), table (table name), pt (partition), len (number of rows), size (storage size in bytes), day (recording date)>.

[0085] In one embodiment, monitor the data table to be detected based on the data volume collection script, the data volume monitoring script, and the data table volume statistics information table to obtain the monitoring results, including: write the data volume collection script and the data volume monitoring script; configure the data volume collection script and the data volume monitoring script into the task flow scheduling system; run the data volume collection script and the data volume monitoring script, and monitor the data table to be detected according to the data table volume statistics information table to obtain the process parameters and the monitoring results.

[0086] In one embodiment, run the data volume collection script and the data volume monitoring script, and monitor the data table to be detected according to the data table volume statistics information table to obtain the process parameters and the monitoring results, including: run the data volume collection script; collect the volume data of the data table to be detected in the preset partition on the preset date based on the data volume collection script, and save the volume data to the data table volume statistics information table; run the data volume monitoring script; monitor the data table to be detected based on the data volume monitoring script and the data table volume statistics information table to obtain the process parameters and the monitoring results.

[0087] Specifically, as Figure 3 shown, S3: Write a data volume collection script A. The function of the data volume collection script A is to collect the volume data of the data table T to be detected in the t partition on the t day, and store this volume data into the Q table (data table volume statistics information table). Among them, the input parameters of the data volume collection script A are (full table name, partition), which can be (T, t) here, and the return parameter is collection success or collection failure.

[0088] For example, Figure 4 As shown, call the hive metastore service interface (metadata storage), read the number of rows and bytes of the pt partition data in the T table (the data table to be tested), and use day (record date), instance (database instance name), db (database name), table (table name), and pt (partition) as conditions to query whether there are records in the Q table (data table volume statistics table). If there are records, delete the existing data and save the records to the Q table; if there are no records, save the records directly to the Q table; if the save is successful, return success, if the save fails, return failure.

[0089] Specifically, Figure 3 As shown, S4: Write a data volume monitoring script B. The function of the volume monitoring script B is to monitor whether there is a problem with the data volume of the t partition of the T table (the data table to be tested) by implementing formulas (1) to (6) and combining the Q table (data table volume statistics table). The calculation process parameters and calculation results are stored in the data quality detection table W for subsequent verification and algorithm optimization. Here, the input parameter of the data volume monitoring script B is (full table name, partition), and the return value is the enumeration value [success, overflow, underflow]. The following is an explanation of formulas (1) to (6) in combination with specific parameters.

[0090] It should be noted that there may be duplicate values ​​in the data of the same table and the same partition in the Q table (data table volume statistics information table). In this case, if the record date day is different, the latest value is taken according to the record date day; if the record date day is also the same, it means that there is a BUG (i.e. a problem) in the data volume collection script A. In this case, you can only clear the abnormal data and re-run the data volume collection script A.

[0091] Specifically, Figure 3 As shown, S5: configure the data volume collection script A and the data volume monitoring script B into the task flow scheduling system, and the corresponding task names are Task-A and Task-B, respectively, and the task dependencies are: Task-T->Task-A->Task-B, that is, Task-T depends on Task-A, and Task-A depends on Task-B. When the downstream task fails, the downstream task is blocked. It should be noted that the dependency relationship of Task-T->Task-A->Task-B is a separate branch and should not affect the original downstream dependency relationship of Task-T. For example, when Task-A and Task-B tasks are not configured, there is a dependency relationship Task-T->Task-N. After Task-A and Task-B tasks are configured, the dependency relationship becomes: Task-T->Task-N

[0092] \

[0093] ->Task-A->Task-B.

[0094] In one embodiment, based on the data volume collection script, the volume data of the preset partition of the data table to be tested on a preset date is collected, and the volume data is saved to the data table volume statistical information table, including: reading the records in the data table volume statistical information table based on the instance, database, table, and preset partition as conditions; selecting the record of the latest date from the records; calculating the volume baseline value of the data table to be tested on day t based on the record of the latest date; calculating the confidence interval of the data volume of the data table to be tested on day t based on the record of the latest date based on the box-and-line method; saving the volume baseline value of the data table to be tested and the confidence interval of the data volume of the data table to be tested on day t to the data table volume statistical information table.

[0095] In one embodiment, the volume reference value of the to-be-detected data table on day t is calculated:

[0096]

[0097] Among them, the above formulas are formula (1) to formula (3), represents the estimated number of rows on day t; Δ represents the average number of bytes per row in the past 7 days; represents the estimated number of bytes on day t; S i Indicates the number of bytes; L i Indicates the number of rows; t-1 means the day before t, t-2 means 2 days before t, and t-7 means 7 days before t.

[0098] In one embodiment, calculating the confidence interval of the data volume on day t of the to-be-detected data table based on the records of the latest date based on the box plot method includes:

[0099] Calculate the maximum value a_max, the upper quartile a3 / 4, the median a2 / 4, and the lower quartile a1 / 4 of the data bytes in the data table to be tested in the past 60 days, and calculate the harmonic upper limit A_max and the harmonic lower limit A_min based on the maximum value a_max, the upper quartile a3 / 4, the median a2 / 4, and the lower quartile a1 / 4 of the minimum value a_min:

[0100] A_max=max(a_max,a3 / 4+a3 / 4-a2 / 4)

[0101] A_min=min(a_min,a1 / 4-(a2 / 4-a1 / 4))

[0102] Calculate the confidence interval of the data volume on day t of the data table to be tested:

[0103] CI = (A_min, A_max)

[0104] Among them, the above formulas are formula (4) to formula (6); CI is the confidence interval of the data volume on day t of the data table to be tested.

[0105] Specifically, Figure 5 As shown, the statistical information of the data table volume is collected from the to-be-detected data table T to the Q table, and the Q table contains the following fields: <instance (database instance name), db (database name), table (table name), pt (partition), len (number of rows), size (storage size in bytes), day (record date)>;

[0106] Calculate the T-table volume benchmark value on day t:

[0107]

[0108] It should be noted that represents the estimated number of rows on day t; Δ represents the average number of bytes per row in the past 7 days; represents the estimated number of bytes on day t; S i Indicates the number of bytes; L i Indicates the number of rows; t-1 indicates the day before t, t-2 indicates the 2 days before t, and t-7 indicates the 7 days before t. For example, assuming t is January 10, 2024, then t-1 is January 9, 2024, and t-2 is January 8, 2024.

[0109] Specifically, in this embodiment, the confidence interval of the data volume on day t is calculated based on the box-line method: the maximum value a_max, the upper 4 quartile a3 / 4, the median a2 / 4, and the lower 4 quartile a1 / 4 of the T-table data bytes in the past 60 days are calculated, and the harmonic upper limit A_max and the harmonic lower limit A_min are calculated according to the maximum value a_max, the upper 4 quartile a3 / 4, the median a2 / 4, and the lower 4 quartile a1 / 4 of the minimum value a_min:

[0110] A_max=max(a_max,a3 / 4+a3 / 4-a2 / 4)

[0111] A_min=min(a_min,a1 / 4-(a2 / 4-a1 / 4))

[0112] Calculate the confidence interval for the data volume on day t in table T:

[0113] CI = (A_min, A_max)

[0114] In one embodiment, the data quality detection table is read based on the alarm service to generate real-time alarm information, including: reading the data quality detection table based on the alarm service to obtain the confidence interval of the data volume on day t of the data table to be detected; comparing the data volume on the current day with the confidence interval; if the data volume size on the current day falls within the confidence interval, the data volume detection on the current day passes; if the data volume on the current day is less than the harmonic lower limit of the confidence interval or greater than the harmonic upper limit of the confidence interval, a real-time alarm information is generated.

[0115] Specifically, Figure 3 As shown, S6: the alarm service reads the data in the W table (data quality detection table) and issues real-time alarms according to the result status and day generation time. The alarm results should be translated accordingly. For example, overflow corresponds to data anomalies, the detection result is overflow, and the preliminary judgment is that there are a large number of null values ​​or the data source access data is incomplete; overflow corresponds to data anomalies, the detection result is overflow, and the preliminary judgment is that there are duplicate data; the alarm results (or real-time alarm information) can be sent to the data person in charge or operation and maintenance personnel through emails, communication software, etc.

[0116] Exemplarily, the data volume of the day can be converted based on the volume baseline value of the data table to be detected and compared with the confidence interval. If the data volume size of the day falls within the CI confidence interval, it is considered that the data volume detection of the day has passed; if the data volume size of the day is less than A_min, it is considered that the data access terminal has missed data or there are a large number of null values, and an alarm is given; if the data volume size of the day is greater than A_max, it is considered that duplicate data exists, and an alarm is given. Compared with the traditional method, the beneficial effects of this embodiment include using the number of rows and the number of data bytes to comprehensively determine whether the data volume is normal, and can also provide a rough problem location. This method is simple and easy to understand and implement.

[0117] This embodiment provides a data volume assessment method based on the box-and-line method, including: determining a data table to be tested; creating a data table volume statistics information table and a data quality detection table; monitoring the data table to be tested based on a data volume collection script, a data volume monitoring script and the data table volume statistics information table to obtain monitoring results; wherein the monitoring results include the volume baseline value of the data table to be tested and the confidence interval of the data volume of the data table to be tested on day t calculated based on the box-and-line method; saving process parameters and the monitoring results to the data quality detection table; reading the data quality detection table based on the alarm service to generate real-time alarm information. In this embodiment, the statistical confidence interval of the data volume is calculated based on the box-and-line method, and combined with the growth model of the data volume of the data table, so as to automatically detect the data volume of the data table, and the detection efficiency is faster and the accuracy is higher. Compared with the traditional method, this embodiment uses the number of rows and the number of data bytes to comprehensively determine whether the data volume is normal, and can also give a rough problem location. This method is simple and easy to understand and implement.

[0118] In addition, an embodiment of the present invention also proposes a storage medium, on which a data volume assessment program based on the box-and-whisker method is stored. When the data volume assessment program based on the box-and-whisker method is executed by a processor, the steps of the data volume assessment method based on the box-and-whisker method as described above are implemented.

[0119] Reference Figure 6 , Figure 6 It is a structural block diagram of an embodiment of a data volume assessment system based on the box-and-whisker method of the present invention.

[0120] like Figure 6 As shown, the data volume evaluation system based on the box-line method includes:

[0121] A determination module 10, used to determine a data table to be tested;

[0122] New module 20 is used to create a new data table volume statistics information table and a data quality detection table;

[0123] A monitoring module 30 is used to monitor the data table to be tested based on a data volume collection script, a data volume monitoring script and the data table volume statistical information table to obtain a monitoring result; the monitoring result includes a volume reference value of the data table to be tested and a confidence interval of the data volume of the data table to be tested on day t calculated based on the box-line method;

[0124] A saving module 40, used for saving the monitoring process parameters and the monitoring results to the data quality detection table;

[0125] The alarm module 50 is used to read the data quality detection table based on the alarm service and generate real-time alarm information.

[0126] This embodiment provides a data volume assessment system based on the box-and-whisker method. In this embodiment, the statistical confidence interval of the data volume is calculated based on the box-and-whisker method, and combined with the growth model of the data volume of the data table, the data volume of the data table is automatically detected, and the detection efficiency is faster and the accuracy is higher. Compared with the traditional method, this embodiment uses the number of rows and the number of data bytes to comprehensively determine whether the data volume is normal, and can also provide a rough problem location. The system is simple and easy to understand and implement.

[0127] It should be noted that, for technical details that are not described in detail in this embodiment of the data volume assessment system based on the box-and-whisker method, reference can be made to the data volume assessment method based on the box-and-whisker method as described above provided in any embodiment of the present invention, and they will not be repeated here.

[0128] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0129] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0130] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0131] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0133] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data volume assessment method based on the boxplot method, characterized in that: include: Determine the data table to be tested; Create a new data table volume statistics table and data quality detection table; Based on the data volume collection script, the data volume monitoring script and the data table volume statistical information table, the data table to be tested is monitored to obtain a monitoring result; wherein the monitoring result includes a volume benchmark value of the data table to be tested and a confidence interval of the data volume of the data table to be tested on day t calculated based on the box-line method; Saving the process parameters and the monitoring results to the data quality detection table; The data quality detection table is read based on the alarm service to generate real-time alarm information.

2. The method according to claim 1, characterized in that The newly created data table volume statistics information table and data quality detection table include: Calling a service interface to read the number of data rows and bytes in a preset partition of the data table to be tested; Use date, database instance name, database name, table name, and preset partition as conditions to query whether records already exist in the data table volume statistics table; If the record already exists, delete the existing data and save the number of rows and bytes of the read data to the data table volume statistics table; If no record exists, the number of rows and bytes of the read data is saved in the data table volume statistics table; Create a new data quality detection table.

3. The method according to claim 1, characterized in that The monitoring of the data table to be tested based on the data volume collection script, the data volume monitoring script and the data table volume statistical information table to obtain the monitoring result includes: Write data volume collection scripts and data volume monitoring scripts; Configuring the data volume collection script and the data volume monitoring script into the task flow scheduling system; Run the data volume collection script and the data volume monitoring script, monitor the data table to be tested according to the data table volume statistical information table, and obtain process parameters and monitoring results.

4. The method according to claim 3, characterized in that The running of the data volume collection script and the data volume monitoring script, monitoring the data table to be tested according to the data table volume statistical information table, and obtaining process parameters and monitoring results, includes: Running the data volume collection script; Based on the data volume collection script, the volume data of the preset partition of the data table to be tested on a preset date is collected, and the volume data is saved to the data table volume statistical information table; Run the data volume monitoring script; The data table to be tested is monitored based on the data volume monitoring script and the data table volume statistical information table to obtain process parameters and monitoring results.

5. The method according to claim 4, characterized in that The step of collecting the volume data of the preset partition of the data table to be tested on a preset date based on the data volume collection script, and saving the volume data to the data table volume statistical information table, includes: Read the records in the data table volume statistics information table based on the instance, database, table, and preset partition as conditions; Selecting the record with the latest date from the records; Calculate the volume reference value of the to-be-detected data table on day t based on the record of the latest date; Calculate the confidence interval of the data volume on day t of the data table to be detected based on the records of the latest date based on the box plot method; The volume benchmark value of the data table to be detected and the confidence interval of the data volume on day t of the data table to be detected are saved in the data table volume statistical information table.

6. The method according to claim 5, characterized in that Calculate the volume benchmark value of the data table to be tested on day t: in, represents the estimated number of rows on day t; Δ represents the average number of bytes per row in the past 7 days; represents the estimated number of bytes on day t; S i Indicates the number of bytes; L i Indicates the number of rows; t-1 means the day before t, t-2 means 2 days before t, and t-7 means 7 days before t.

7. The method according to claim 5, characterized in that The calculating the confidence interval of the data volume on day t of the to-be-detected data table according to the records of the latest date based on the box plot method includes: Calculate the maximum value a_max, the upper quartile a3 / 4, the median a2 / 4, and the lower quartile a1 / 4 of the data bytes in the data table to be tested in the past 60 days, and calculate the harmonic upper limit A max and the harmonic lower limit A_min respectively based on the maximum value a max, the upper quartile a3 / 4, the median a2 / 4, and the lower quartile a1 / 4 of the minimum value a min: A_max=max(a_max,a3 / 4+a3 / 4-a2 / 4) A_min=min(a_min,a1 / 4-(a2 / 4-a1 / 4)) Calculate the confidence interval of the data volume on day t of the data table to be tested: CI = (A_min, A_imax) Among them, CI is the confidence interval of the data volume on day t in the data table to be tested.

8. The method according to claim 7, characterized in that The alarm service-based method of reading the data quality detection table and generating real-time alarm information includes: Read the data quality detection table based on the alarm service to obtain the confidence interval of the data volume on day t of the data table to be detected; Compare the data volume of the day with the confidence interval; If the data volume size of the day falls within the confidence interval, the data volume detection of the day passes; If the data volume for the day is less than the harmonic lower limit of the confidence interval or greater than the harmonic upper limit of the confidence interval, a real-time alarm message is generated.

9. A data volume assessment system based on the boxplot method, characterized in that: include: A determination module, used for determining a data table to be tested; Create a new module to create a new data table volume statistics table and data quality detection table; A monitoring module, used to monitor the data table to be tested based on a data volume collection script, a data volume monitoring script and the data table volume statistical information table to obtain a monitoring result; the monitoring result includes a volume benchmark value of the data table to be tested and a confidence interval of the data volume of the data table to be tested on day t calculated based on the box plot method; A saving module, used for saving the monitoring process parameters and the monitoring results to the data quality detection table; The alarm module is used to read the data quality detection table based on the alarm service and generate real-time alarm information.

10. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a box-and-whisker method-based data volume assessment program stored in the memory and executable on the processor, wherein the box-and-whisker method-based data volume assessment program is configured to implement the box-and-whisker method-based data volume assessment method as described in any one of claims 1 to 8.

Images