Real-time acquisition method for database

Abstract

The invention relates to the technical field of data acquisition, in particular to a real-time acquisition method for a database, which comprises the following steps: determining a database state based on an abnormal frequency flow coefficient and a compatible influence coefficient, and determining an adjustment and optimization setting mode according to the database state; according to the high-frequency execution coefficient and the sensitive bottleneck threshold, determining the category of an adjustment and optimization statement, and performing adjustment and optimization on the high-frequency influence statement; determining a data division mode of the to-be-analyzed change log according to the tuning comparison coefficient and the dynamic load intensity to obtain a plurality of data fragments; determining a data fragment set according to the distribution adaptation degree, and determining a distribution mode of each data fragment set according to the fragment difference degree and the pre-delay coefficient; determining an optimization mode according to the delay coefficient and the adjustment comparison coefficient; the change data acquisition efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a real-time acquisition method for a database. Background Art

[0002] With the complexity of business systems and the exponential growth of data volume, data changes in databases face severe problems such as acquisition lag, synchronization delay, and insufficient real-time performance, resulting in downstream systems being unable to reflect business dynamic changes in a timely and accurate manner. Therefore, how to improve the efficiency of changed data acquisition is a technical problem that needs to be solved urgently by those skilled in the art.

[0003] Chinese Patent Publication No. CN114547045A discloses a data acquisition and storage method based on a real-time database, including: S1. Start an antivirus software to scan the database. After confirming that no rogue software or virus is implanted, start the data of the data source real-time database device; S2. Regularly collect data through a java project in the form of a restful interface; S3. After collecting the interface data, store it in a real-time data table, an N-hour data table, and a historical data table respectively. It can be seen that the above technical solution has the following problems: It is impossible to accurately optimize the statements in the database. In the face of complex queries or high-concurrency scenarios, it is difficult to break through the database performance bottleneck, thereby resulting in poor changed data acquisition efficiency. Summary of the Invention

[0004] To this end, the present invention provides a real-time acquisition method for a database to overcome the problems in the prior art that it is impossible to accurately optimize the statements in the database, and it is difficult to break through the database performance bottleneck in the face of complex queries or high-concurrency scenarios, thereby resulting in poor changed data acquisition efficiency.

[0005] To achieve the above object, the present invention provides a real-time acquisition method for a database, including: Determine the database state based on the abnormal frequency flow coefficient and the compatibility impact coefficient, and determine the tuning setting method according to the database state as trigger-based tuning according to the trigger impact coefficient or cyclic tuning according to the performance load coefficient; Determine the type of tuning statement according to the high-frequency execution coefficient and the sensitive bottleneck threshold, and optimize the high-frequency impact statement; Determine the data partitioning method of the change log to be analyzed according to the tuning comparison coefficient and the dynamic load intensity to obtain several data segments. The data partitioning method is uniform segment partitioning according to the evaluation deviation index, or determining the dynamic partitioning method according to the attribute fixity and the range coordination coefficient; The dynamic partitioning method is association partitioning according to the attribute similarity and the attribute representation value, or multi-dimensional partitioning according to the impact similarity and the hash similarity; Determine the set of data segments according to the distribution adaptability, and determine the distribution method of each set of data segments as multi-path parallel distribution or single-path interval distribution according to the segment difference degree and the pre-delay coefficient; Determine the optimization method as adjusting the transmission interval according to the delay coefficient and the adjustment comparison coefficient, or determine the priority adjustment method according to the priority pre-adjustment coefficient, and the priority adjustment method is partial priority replacement or overall priority replacement.

[0006] Further, if the database state is that the abnormal frequency coefficient is greater than or equal to the preset abnormal frequency coefficient or the compatibility impact coefficient is greater than or equal to the preset compatibility impact coefficient, the tuning setting method is cyclic tuning according to the performance load coefficient.

[0007] Further, if the database state is that the abnormal frequency coefficient is less than the preset abnormal frequency coefficient and the compatibility impact coefficient is less than the preset compatibility impact coefficient, the tuning setting method is trigger tuning according to the trigger impact coefficient.

[0008] Further, determine the type of tuning statement according to the high-frequency execution coefficient and the sensitive bottleneck threshold, and the types of tuning statements include: High-frequency impact statements with a high-frequency execution coefficient greater than or equal to the preset high-frequency execution coefficient or a sensitive bottleneck threshold greater than or equal to the preset sensitive bottleneck threshold; Low-frequency impact statements with a high-frequency execution coefficient less than the preset high-frequency execution coefficient and a sensitive bottleneck threshold less than the preset sensitive bottleneck threshold.

[0009] Further, determine the data partitioning method of the change log to be analyzed according to the tuning comparison coefficient and the dynamic load intensity, including: If the tuning comparison coefficient is less than the preset tuning comparison coefficient and the dynamic load intensity is less than the preset dynamic load intensity, the data partitioning method is uniform segment partitioning according to the evaluation deviation index; If the tuning comparison coefficient is greater than or equal to the preset tuning comparison coefficient or the dynamic load intensity is greater than or equal to the preset dynamic load intensity, the data partitioning method is to determine the dynamic partitioning method according to the attribute fixity and the range coordination coefficient.

[0010] Further, if the attribute fixity is greater than or equal to the preset attribute fixity and the range coordination coefficient is greater than or equal to the preset range coordination coefficient, the dynamic partitioning method is correlation partitioning according to the attribute similarity and the attribute representation value.

[0011] Further, if the attribute fixity is less than the preset attribute fixity or the range coordination coefficient is less than the preset range coordination coefficient, the dynamic partitioning method is multi-dimensional partitioning according to the impact similarity and the hash similarity.

[0012] Further, determine the distribution method of each data segment set according to the segment difference degree and the pre-delay coefficient, including: For a single data segment set, If the segment difference degree is greater than or equal to the preset segment difference degree or the pre-delay coefficient is greater than or equal to the preset pre-delay coefficient, the distribution method is multi-path parallel distribution; If the segment difference degree is less than the preset segment difference degree and the pre-delay coefficient is less than the preset pre-delay coefficient, the segment distribution method is single-path interval distribution.

[0013] Further, if the delay coefficient is greater than or equal to the preset delay coefficient or the adjustment comparison coefficient is greater than or equal to the preset adjustment comparison coefficient, the optimization method is to determine the priority adjustment method according to the priority pre-adjustment coefficient; If the priority pre-adjustment coefficient is less than the preset priority pre-adjustment coefficient, the priority adjustment method is partial priority replacement; If the priority pre-adjustment coefficient is greater than or equal to the preset priority pre-adjustment coefficient, the priority adjustment method is overall priority replacement.

[0014] Further, if the delay coefficient is less than the preset delay coefficient and the adjustment comparison coefficient is less than the preset adjustment comparison coefficient, the optimization method is to perform a reduction adjustment on the set transmission interval; The reduction value of the set transmission interval has a negative correlation with the anomaly evaluation coefficient.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows. In the technical solution of the present invention, the database state is determined based on the abnormal frequency flow coefficient and the compatibility influence coefficient. The abnormal situation of the database is effectively reflected by the abnormal frequency flow coefficient and the compatibility influence coefficient. Then, different tuning setting methods are adaptively selected according to the database state, making the selection of the tuning setting method more in line with the actual application scenario, helping to accurately locate problems and improve the tuning efficiency. The importance of SQL statements is effectively reflected by the high-frequency execution coefficient and the sensitive bottleneck threshold, and then the high-frequency impact statements are tuned. While improving the data processing efficiency, redundant or repeated data changes can be reduced, and thus the change data collection efficiency can be improved.

[0016] Further, in the present invention, the tuning comparison situation and the load situation are effectively reflected by the tuning comparison coefficient and the dynamic load intensity. Then, different data partitioning methods are adaptively selected according to the tuning comparison coefficient and the dynamic load intensity. Uniform segment partitioning according to the evaluation deviation index can ensure the balance of data segments, reduce the sharding overhead, and avoid resource waste. Determining the dynamic partitioning method according to the attribute fixity and the range coordination coefficient can reduce data redundancy and improve the analysis efficiency.

[0017] Furthermore, in the present invention, the potential association status among data in the database is effectively reflected by the attribute fixation degree and the range coordination coefficient, and then different dynamic partitioning methods are adaptively selected according to the attribute fixation degree and the range coordination coefficient, so that the dynamic partitioning method can adapt to data changes and improve the efficiency of collecting changed data.

[0018] Furthermore, in the present invention, the distribution method of each data fragment set is determined according to the fragment difference degree and the pre-delay coefficient. Through multi-path parallel distribution, the network bandwidth resources can be fully utilized, significantly improving the data transmission rate and system throughput. Through single-path interval distribution, path congestion can be avoided, ensuring the orderly transmission of data, reducing transmission interruptions caused by path failures, and facilitating the timely transmission of data to the change analysis node for data change analysis, thereby improving the efficiency of collecting changed data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic diagram of the real-time acquisition method for the database of the present invention; Figure 2 is a flowchart of determining the tuning setting method according to the database status of the present invention; Figure 3 is a flowchart of determining the data partitioning method according to the tuning comparison coefficient and the dynamic load intensity of the present invention; Figure 4 is a flowchart of determining the fragment distribution method according to the fragment difference degree and the pre-delay coefficient of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] In order to make the objectives and advantages of the present invention clearer, the present invention will be further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0021] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and do not limit the protection scope of the present invention.

[0022] It should be noted that in the description of the present invention, the terms indicating directions or positional relationships such as "upper", "lower", "left", "right", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings. This is only for convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0023] In addition, it should be noted that in the description of the present invention, unless otherwise clearly defined and limited, the terms "installation", "connection", and "coupling" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0024] Please refer to Figures 1 to 4 as shown, the present invention provides a real-time acquisition method for a database, including: Determining the database status based on the abnormal frequency coefficient and the compatibility impact coefficient, and determining the tuning setting method according to the database status as trigger-based tuning according to the trigger impact coefficient or cyclic tuning according to the performance load coefficient; Determining the tuning statement category according to the high-frequency execution coefficient and the sensitive bottleneck threshold, and tuning the high-frequency impact statements; Determining the data partitioning method of the change log to be analyzed according to the tuning comparison coefficient and the dynamic load intensity to obtain several data segments. The data partitioning method is uniform segment partitioning according to the evaluation deviation index, or determining the dynamic partitioning method according to the attribute fixity and the range coordination coefficient; The dynamic partitioning method is correlation partitioning according to the attribute similarity and the attribute representation value, or multi-dimensional partitioning according to the impact similarity and the hash similarity; Determining the data segment set according to the distribution adaptability, and determining the distribution method of each data segment set as multi-path parallel distribution or single-path interval distribution according to the segment difference degree and the pre-delay coefficient; Determining the optimization method as adjusting the transmission interval according to the delay coefficient and the adjustment comparison coefficient, or determining the priority adjustment method according to the priority pre-adjustment coefficient. The priority adjustment method is partial priority replacement or overall priority replacement.

[0025] The application scenario of the present invention is the collection of database change data. In the present invention, the database contains several SQL statements, and each SQL statement corresponds to a log. The SQL statement is responsible for defining the read and write operations on the database, while the log records the detailed information of the read and write operations. The log records the execution process and results of the SQL statement. The log includes, but is not limited to, the execution timestamp of the SQL statement, the specific content of the SQL statement, and the execution result of the SQL statement. The change log is the log of insert, update, or delete operations captured in the database by Debezium. Each single change log contains several timestamps, and each timestamp corresponds to a log interval, indicating that all log lines within the log interval perform insert, update, or delete operations at this time point. Each single log interval contains several log lines, which is easy for those skilled in the art to understand and will not be elaborated here specifically; In the present invention, several historical records are correspondingly set. Any one of the historical records records at least the sub-abnormal frequency flow value, abnormal frequency flow coefficient, compatibility impact coefficient, trigger impact coefficient, high-frequency execution coefficient, and sensitive bottleneck threshold, etc. in the historical process of at least one database change data collection. And each historical record corresponds to a qualified mark, and the qualified mark records whether the process of database change data collection meets the user's requirements. The qualified mark can be manually recorded. It can be understood that the user can determine whether the process of database change data collection meets the requirements according to the self-set indicators. The self-set indicators can be, but are not limited to, the transmission index, which will not be elaborated here. Among them, the transmission index is the duration of transmitting the change log to the change analysis node; The present invention includes several transmission paths and a change analysis node. The data fragment can be transmitted to the change analysis node through the transmission path. The change analysis node can extract key information from the change log, identify potential risks, and provide data support for subsequent decision-making. The specific extraction method is easy for those skilled in the art to understand and will not be elaborated here specifically; When optimizing high-frequency impact statements, the methods that the user can adopt include, but are not limited to, delaying the writing of non-critical data, using temporary tables or intermediate tables, and optimizing indexes. The user can select according to actual needs. This is a common technical means for those skilled in the art and will not be elaborated here specifically; Determining the data fragment set according to the distribution adaptability includes: performing combined analysis on each data fragment. When performing combined analysis on a single data fragment, this data fragment is recorded as the target data fragment, and the data fragments that are not recorded in the data fragment set except the target data fragment are recorded as reference data fragments. The set of reference data fragments and the target data fragment whose distribution adaptability to the target data fragment is greater than the preset distribution adaptability is recorded as a data fragment set, and continue to perform combined analysis on each data fragment that is not recorded in the data fragment set until all data fragments are recorded in the data fragment set, then stop the combined analysis; It should be noted that the transmission priority coefficient of a single data segment set is positively correlated with the transmission requirement coefficient corresponding to the data segment set. The larger the transmission priority coefficient of the data segment set, the more preferential the transmission order; The confirmation method of the transmission requirement coefficient is as follows: for a single data segment set, denote the data segment set as the target set, denote the other data segment sets outside the target set as the reference sets, denote the keywords appearing in each data segment in the target set as the reference words, and the transmission requirement coefficient corresponding to the target set is the average value of the segment reference values corresponding to each reference word. The segment reference value corresponding to a single reference word is the number of reference sets in which the reference word appears; The confirmation method of the distribution adaptability is as follows: for any two data segments, the distribution adaptability corresponding to the two data segments = 1 - (the absolute value of the difference between the sub-distribution coefficients corresponding to the two data segments / the larger value of the sub-distribution coefficients corresponding to the two data segments). The sub-distribution coefficient corresponding to a single data segment = the number of different keywords contained in the data segment / the number of log lines contained in the data segment. The value of the preset distribution adaptability can be determined by the user according to the actual application scenario. The greater the user's demand for data distribution efficiency, the larger the value of the preset distribution adaptability. Provide a value of the preset distribution adaptability, and the preset distribution adaptability is 70%;

[0026] Specifically, if the database state is that the abnormal frequency flow coefficient is greater than or equal to the preset abnormal frequency flow coefficient or the compatibility impact coefficient is greater than or equal to the preset compatibility impact coefficient, the tuning setting method is to perform cyclic tuning according to the performance load coefficient.

[0027] Among them, in the present invention, a continuous cyclic tuning setting period is set. The database state is determined once at the end of each tuning setting period. The duration of the tuning setting period can be set according to the user's needs. The greater the user's demand for tuning accuracy, the smaller the duration of the tuning setting period. Provide a value of the tuning setting period, and the tuning setting period is 1d; denote the tuning setting period adjacent to and before the current tuning setting period as the target tuning setting period; The database state includes the first database state and the second database state. The first database state is that the abnormal frequency flow coefficient is greater than or equal to the preset abnormal frequency flow coefficient or the compatibility impact coefficient is greater than or equal to the preset compatibility impact coefficient. The second database state is that the abnormal frequency flow coefficient is less than the preset abnormal frequency flow coefficient and the compatibility impact coefficient is less than the preset compatibility impact coefficient; The abnormal frequency flow coefficient is the average value of the sub - abnormal frequency flow values corresponding to each historical record. The way to confirm the sub - abnormal frequency flow value is as follows: for a single historical record, mark this historical record as the target historical record, mark the other historical records except the target historical record as reference historical records, and mark the average value of the abnormal interaction degrees corresponding to the target historical record and each reference historical record as the sub - abnormal frequency flow value corresponding to the target historical record. The way to confirm the abnormal interaction degree is as follows: for any two historical records, respectively detect the hash values of the data segments corresponding to each changed data in the two historical records. The abnormal interaction degree corresponding to the two historical records = the number of the same hash values in one historical record and the other historical record / the larger value of the number of data segments corresponding to each changed data in the two historical records; for the hash value corresponding to a single data segment, the user can map this data segment to a string through a hash function, and the hash function includes but is not limited to MD5, SHA - 1, and SHA - 256. The user can select according to actual needs without specific restrictions; The compatibility impact coefficient = (the first impact value - the second impact value) / the first impact value. Mark the historical records with sub - abnormal frequency flow values greater than or equal to the preset sub - abnormal frequency flow value as the first historical records, and mark the historical records with sub - abnormal frequency flow values less than the preset sub - abnormal frequency flow value as the second historical records. The first impact value is the average value of the capture times corresponding to each first historical record, and the second impact value is the average value of the capture times corresponding to each second historical record. The capture time corresponding to a single historical record is the time used by the change analysis node after transmitting the data segment to the change analysis node in this historical record; For the values of the preset abnormal frequency flow coefficient, the preset compatibility impact coefficient, and the preset sub - abnormal frequency flow value, the user can determine them according to the actual application scenario. The greater the user's demand for improving the efficiency of capturing changed data, the greater the values of the preset abnormal frequency flow coefficient and the preset compatibility impact coefficient. Provide a way to determine the values of the preset abnormal frequency flow coefficient and the preset compatibility impact coefficient. Detect the historical records of the user's cyclic tuning according to the performance load coefficient, and mark the average value of the abnormal frequency flow coefficients corresponding to the historical records that can meet the user's needs as the preset abnormal frequency flow coefficient, and mark the average value of the compatibility impact coefficients corresponding to the historical records that can meet the user's needs as the preset compatibility impact coefficient. The greater the value of the preset sub - abnormal frequency flow value, the greater the user's demand to determine the historical record as the second historical record. Provide a value of the preset sub - abnormal frequency flow value of 40%; Cyclic tuning according to the performance load coefficient includes: setting a continuously cycling tuning period, and performing a tuning on the high - frequency impact statements in the database at the end of each tuning period. The duration of a single tuning period has a negative correlation with the performance load coefficient; Performance load factor = (abnormal frequency flow coefficient + compatibility impact coefficient) × the number of processing requests of the database within the target tuning setting period, where the number of processing requests of the database within the target tuning setting period is the total number of operations such as queries, updates, inserts, deletes, etc. received by the database from the application, user, or other systems within the target tuning setting period.

[0028] Specifically, if the database status is that the abnormal frequency flow coefficient is less than the preset abnormal frequency flow coefficient and the compatibility impact coefficient is less than the preset compatibility impact coefficient, the tuning setting method is trigger-based tuning according to the trigger impact coefficient.

[0029] Among them, trigger-based tuning according to the trigger impact coefficient includes: when the trigger impact coefficient is greater than the preset trigger impact coefficient, tuning the high-frequency impact statements in the database; Trigger impact coefficient = sub-request coefficient corresponding to the current moment × surge coefficient. The sub-request coefficient corresponding to a single moment is the number of operations such as queries, updates, inserts, deletes, etc. received from the application, user, or other systems at that moment. Surge coefficient = (sub-request coefficient corresponding to the current moment - average value of sub-request coefficients corresponding to each reference moment before the current moment) / average value of sub-request coefficients corresponding to each reference moment before the current moment; The reference moment is set by the user himself. A method for setting the reference moment is provided. According to the order of the current tuning setting period from early to late, every 1 minute is recorded as a reference moment; For the value of the preset trigger impact coefficient, the user can determine it according to the actual application scenario. The greater the user's demand for improving the change data capture efficiency, the smaller the value of the preset trigger impact coefficient. A value of the preset trigger impact coefficient is provided. Detect the historical records of the user's trigger-based tuning according to the trigger impact coefficient, and record the average value of the trigger impact coefficients corresponding to the moments when tuning is performed in the historical records that can meet the user's needs as the preset trigger impact coefficient.

[0030] Specifically, determine the type of tuning statement according to the high-frequency execution coefficient and the sensitive bottleneck threshold. The types of tuning statements include: High-frequency impact statements where the high-frequency execution coefficient is greater than or equal to the preset high-frequency execution coefficient or the sensitive bottleneck threshold is greater than or equal to the preset sensitive bottleneck threshold; Low-frequency impact statements where the high-frequency execution coefficient is less than the preset high-frequency execution coefficient and the sensitive bottleneck threshold is less than the preset sensitive bottleneck threshold.

[0031] Among them, the high-frequency execution coefficient corresponding to a single SQL statement = the number of times the SQL statement is executed within the target tuning setting period / the sum of the number of times all SQL statements are executed within the target tuning setting period; Executing an SQL statement is to interact with the database and perform operations such as updating, inserting, or deleting data; Sensitive bottleneck threshold = Execution time × Number of scanned rows. The execution time corresponding to a single SQL statement is the total time from the start of execution of the SQL statement to the completion of execution. The number of scanned rows corresponding to a single SQL statement is the number of row records in the underlying data table or index actually read or checked during the execution of the SQL statement. The number of scanned rows is viewed through MySQL.

[0032] For the values of the preset high-frequency execution coefficient and the preset sensitive bottleneck threshold, the user can determine them according to the actual application scenario. The greater the user's demand for improving the efficiency of changed data collection, the smaller the values of the preset high-frequency execution coefficient and the preset sensitive bottleneck threshold. Provide a way to determine the values of the preset high-frequency execution coefficient and the preset sensitive bottleneck threshold. Denote the average value of the high-frequency execution coefficients corresponding to the high-frequency impact statements in the historical records that can meet the user's needs as the preset high-frequency execution coefficient, and denote the average value of the sensitive bottleneck thresholds corresponding to the high-frequency impact statements in the historical records that can meet the user's needs as the preset sensitive bottleneck threshold.

[0033] Specifically, determine the data partitioning method for the change log to be analyzed according to the tuning comparison coefficient and the dynamic load intensity, including: If the tuning comparison coefficient is less than the preset tuning comparison coefficient and the dynamic load intensity is less than the preset dynamic load intensity, the data partitioning method is to perform uniform segment partitioning according to the evaluation deviation index; If the tuning comparison coefficient is greater than or equal to the preset tuning comparison coefficient or the dynamic load intensity is greater than or equal to the preset dynamic load intensity, the data partitioning method is to determine the dynamic partitioning method according to the attribute fixity and the range coordination coefficient.

[0034] Among them, the change log to be analyzed is the change log captured by Debezium at the current moment; Tuning comparison coefficient = Number of high-frequency impact statements at the current moment / Number of low-frequency impact statements at the current moment; Dynamic load intensity = Log data volume + Log append frequency. The log data volume is the number of change logs to be analyzed at the current moment, and the log append frequency is the number of change logs to be analyzed that write new log records to the end of the log at the current moment; For the values of the preset tuning comparison coefficient and the preset dynamic load intensity, the user can determine them according to the actual application scenario. The greater the values of the preset tuning comparison coefficient and the preset dynamic load intensity, the greater the user's demand for uniform segment partitioning according to the evaluation deviation index. Provide a way to determine the values of the preset tuning comparison coefficient and the preset dynamic load intensity. Detect the historical records of the user's uniform segment partitioning according to the evaluation deviation index, and denote the average value of the tuning comparison coefficients corresponding to the historical records that can meet the user's needs as the preset tuning comparison coefficient, and denote the average value of the dynamic load intensities corresponding to the historical records that can meet the user's needs as the preset dynamic load intensity; Uniform segment division is performed according to the evaluation deviation index, including: evenly dividing the change log to be analyzed into n1 data segments, where n1 has a negative correlation with the evaluation deviation index; when performing the even division of the change log to be analyzed, starting from the initial log line of the change log to be analyzed, the log lines are sequentially assigned to the data segments in the order from front to back. The number of log lines contained in a single data segment is the smallest integer less than or equal to k1, and k1 = the number of log lines contained in the change log to be analyzed / n1. It should be noted that if k1 is not an integer, the last data segment will contain the remaining log lines; a single change log to be analyzed contains several lines, and each line of the change log to be analyzed is denoted as a log line. The evaluation deviation index = the tuning comparison coefficient × the dynamic load intensity.

[0035] Specifically, if the attribute fixity is greater than or equal to the preset attribute fixity and the range coordination coefficient is greater than or equal to the preset range coordination coefficient, the dynamic division method is to perform associated division according to the attribute similarity and the attribute representation value.

[0036] Among them, the confirmation method of the attribute fixity is as follows: for a single change log to be analyzed, denote this change log to be analyzed as the target log, denote the keywords with a frequency coefficient greater than the preset frequency coefficient in the target log as frequency words, and denote the average value of the distribution reference values corresponding to each frequency word as the attribute fixity corresponding to the target log. For a single frequency word, denote this frequency word as the frequency word to be analyzed, denote the log lines containing the frequency word to be analyzed as analysis lines, and denote the average value of the sub-distribution coefficients corresponding to each analysis line as the distribution reference value corresponding to the frequency word to be analyzed. The sub-distribution coefficient corresponding to a single analysis line is the average value of the line intervals from this analysis line to the other analysis lines, and the line interval corresponding to any two analysis lines is the number of log lines between the two analysis lines. The frequency coefficient corresponding to a single keyword is the number of times this keyword appears in the target log. The value of the preset frequency coefficient can be determined by the user according to the actual application scenario. The smaller the value of the preset frequency coefficient, the greater the user's need to determine the keyword as a frequency word. Provide a value of the preset frequency coefficient, and denote the average value of the frequency coefficients corresponding to each frequency word in the historical records that can meet the user's needs as the preset frequency coefficient. The confirmation method of the range coordination coefficient is as follows: for a single change log to be analyzed, the range coordination coefficient corresponding to this change log to be analyzed = 1 - [the standard deviation of the operation time of the log intervals corresponding to each timestamp in this change log to be analyzed × (the time length between the earliest time and the latest time corresponding to each timestamp contained in this change log to be analyzed / the number of timestamps contained in this change log to be analyzed)], and the operation time of the log interval corresponding to a single timestamp is the duration of the insert, update, or delete operation for the log interval corresponding to this timestamp. For the values of the preset attribute fixity and the preset range coordination coefficient, the user can determine them according to the actual application scenario. The smaller the values of the preset attribute fixity and the preset range coordination coefficient are, the greater the user's need to perform association division based on attribute similarity and attribute representation value. Provide a method for determining the values of the preset attribute fixity and the preset range coordination coefficient. Detect the historical records of the user performing association division based on attribute similarity and attribute representation value, and record the average value of the attribute fixity corresponding to the historical records that can meet the user's needs as the preset attribute fixity, and record the average value of the range coordination coefficient corresponding to the historical records that can meet the user's needs as the preset range coordination coefficient; Performing association division based on attribute similarity and attribute representation value includes: for a single change log to be analyzed, evenly divide the change log to be analyzed into n2 log segments, where n2 has a positive correlation with the attribute representation value corresponding to the change log to be analyzed. Perform association analysis on each log segment. When performing association analysis on a single log segment, denote the log segment as the target log segment, denote the log segments of the log paragraphs that have not been included in the data segment except the target log segment as reference log segments, and denote the set of each reference log segment and the target log segment whose attribute similarity to the target log segment is greater than the preset attribute similarity as a data segment, and continue to perform association analysis on the log paragraphs that have not been included in the data segment until all log paragraphs are included in the data segment and then stop the association analysis; When evenly dividing the change log to be analyzed into n2 log segments, starting from the initial log line of the change log to be analyzed, sequentially allocate the log lines to the log segments in the order from front to back. The number of log lines included in a single log segment is the smallest integer less than or equal to k2, where k = the number of log lines included in the change log to be analyzed / n2. It should be noted that if k2 is not an integer, the last log segment will include the remaining log lines; Attribute representation value = attribute fixity × range coordination coefficient. The method for confirming attribute similarity is that for any two log segments, attribute similarity = the number of keywords that exist in both log segments / the time length between the earliest time and the latest time corresponding to each time stamp of the two log segments; For the value of the preset attribute similarity, the user can determine it according to the actual application scenario. The greater the user's need to improve the accuracy of data segment division, the greater the value of the preset attribute similarity. Provide a method for determining the value of the preset attribute similarity. Detect the historical records of performing association division based on attribute similarity and attribute representation value, and record the average value of the reference attribute similarity corresponding to each data segment in the historical records that can meet the user's needs as the preset attribute similarity. The reference attribute similarity corresponding to a single data segment is the attribute similarity corresponding to any two log segments in the data segment.

[0037] Specifically, if the attribute fixity is less than the preset attribute fixity or the range coordination coefficient is less than the preset range coordination coefficient, the dynamic partitioning method is to perform multi-dimensional partitioning based on the impact similarity and the hash similarity.

[0038] Among them, performing multi-dimensional partitioning based on the impact similarity and the hash similarity includes: for a single change log to be analyzed, performing partitioning analysis on each log line of the change log to be analyzed. When performing partitioning analysis on a single log line, this log line is denoted as the target log line, the log lines other than the target log line that have not been recorded into data segments are denoted as reference log lines, and the set of each reference log line and the target log line whose impact similarity with the target log line is greater than the preset impact similarity and the hash similarity is greater than the preset hash similarity is denoted as a data segment, and continue to perform partitioning analysis on the log lines that have not been recorded into data segments until all log lines are recorded into data segments and then stop the partitioning analysis; The confirmation method of the impact similarity is that for any two log lines, the respective keywords corresponding to the two log lines are denoted as the first keyword and the second keyword, and the average value of the co-occurrence coefficients corresponding to each first keyword is denoted as the impact similarity corresponding to the two log lines. Detect the historical records of performing multi-dimensional partitioning based on the impact similarity and the hash similarity, and denote the data segments in which the first keyword appears in the historical records that can meet the user's needs as reference segments. The co-occurrence coefficient corresponding to a single first keyword = the total amount of the second keywords that appear in all reference segments / the number of reference segments; The confirmation method of the hash similarity is that for any two log lines, the hash similarity = the number of identical characters in the strings corresponding to the two log lines / the number of characters contained in the string corresponding to a single log line. The string corresponding to a single log line is the string mapped by the user through the MD5 hash function for this log line; The values of the preset impact similarity and the preset hash similarity can be determined by the user according to the actual application scenario. The greater the user's demand for improving the partitioning accuracy of data segments, the greater the values of the preset impact similarity and the preset hash similarity. Provide a value of the preset impact similarity and the preset hash similarity. The preset hash similarity is 70%. Detect the historical records of the user performing multi-dimensional partitioning based on the impact similarity and the hash similarity, and denote the average value of the reference impact similarities corresponding to each data segment in the historical records that can meet the user's needs as the preset impact similarity. The reference impact similarity corresponding to a single data segment is the impact similarity corresponding to any two log lines in this data segment.

[0039] Specifically, determining the distribution method of each data segment set according to the segment difference degree and the pre-delay coefficient includes: For a single data segment set, If the segment difference degree is greater than or equal to the preset segment difference degree or the pre-delay coefficient is greater than or equal to the preset pre-delay coefficient, the distribution method is multi-path parallel distribution; If the segment difference degree is less than the preset segment difference degree and the pre-delay coefficient is less than the preset pre-delay coefficient, the segment distribution method is single-path interval distribution.

[0040] Among them, the segment difference degree corresponding to a single data segment set is the standard deviation of the segment reference values corresponding to the data segments included in the data segment set, and the segment reference value corresponding to a single data segment is the number of log lines included in the data segment; The pre-delay coefficient corresponding to a single data segment set = the standard deviation of the available bandwidths corresponding to each transmission path / the storage reference value corresponding to the data segment set. The available bandwidth corresponding to a single transmission path is the currently available bandwidth on the transmission path, with the unit of Mbps. The storage reference value corresponding to a single data segment set is the sum of the memories of the data segments in the data segment set, with the unit of MB; For the values of the preset segment difference degree and the preset pre-delay coefficient, the user can determine them according to the actual application scenario. The smaller the values of the preset segment difference degree and the preset pre-delay coefficient, the greater the user's need for multi-path parallel distribution. Provide a set of values for the preset segment difference degree and the preset pre-delay coefficient, detect the historical records of the user's multi-path parallel distribution, and record the average value of the segment difference degrees corresponding to the historical records that can meet the user's needs as the preset segment difference degree, and record the average value of the pre-delay coefficients corresponding to the historical records that can meet the user's needs as the preset pre-delay coefficient; Multi-path parallel distribution includes: using the transmission paths with a path effectiveness coefficient greater than the preset path effectiveness coefficient and a transmission skew coefficient less than the preset transmission skew coefficient as the selected paths, and determining the distribution data volume of each selected path based on the available bandwidth; The method for confirming the path effectiveness coefficient is as follows: for a single transmission path, it can be understood that the present invention collects change data in real time. Therefore, on each transmission path, data segments will be continuously transmitted to each transmission path. For a single transmission path, the data segments that have been distributed to the transmission path and are being transmitted on the transmission path and have not been transmitted to the change analysis node are recorded as the distributed segments, and the data segments to be distributed at the current moment are recorded as the to-be-distributed segments. The path effectiveness coefficient = 1 / the number of the same keywords in the distributed segments and the to-be-distributed segments; The method for confirming the transmission skew coefficient is as follows: for a single transmission path, the transmission skew coefficient = the number of distributed segments in the transmission path at the current moment / the total number of distributed segments in each transmission path at the current moment; For the values of the preset path effectiveness coefficient and the preset transmission skew coefficient, the user can determine them according to the actual application scenario. The greater the user's demand for improving data distribution efficiency, the larger the value of the preset path effectiveness coefficient and the smaller the value of the preset transmission skew coefficient. Provide a method for determining the values of the preset path effectiveness coefficient and the preset transmission skew coefficient. Detect the historical records of the user's multi-path parallel distribution, and record the average value of the path effectiveness coefficients corresponding to each selected path in the historical records that can meet the user's needs as the preset path effectiveness coefficient, and record the average value of the transmission skew coefficients corresponding to each selected path in the historical records that can meet the user's needs as the preset transmission skew coefficient; The data volume to be distributed corresponding to a single selected path = (the available bandwidth corresponding to this selected path / the sum of the available bandwidths corresponding to each selected path) × the number of data segments included in a single data segment set; The data segments to be distributed by a single selected path can be selected by the user, as long as they can meet the data volume to be distributed corresponding to this selected path, and there is no specific limit; Single-path interval distribution includes: when allocating a single data segment set, select the transmission path with the largest path evaluation coefficient as the selected path, and determine the interval distribution method of each data segment in this data segment set according to the channel transmission difficulty value; If the channel transmission difficulty value is greater than or equal to the preset channel transmission difficulty value, the interval distribution method is single-segment interval transmission; If the channel transmission difficulty value is less than the preset channel transmission difficulty value, the interval distribution method is multi-segment interval transmission; For a single transmission path, the path evaluation coefficient corresponding to this transmission path = the path effectiveness coefficient corresponding to this transmission path - the transmission skew coefficient corresponding to this transmission path, and the channel transmission difficulty value = the path evaluation coefficient corresponding to the selected path - the average value of the path evaluation coefficients corresponding to other transmission paths outside the selected path, For the value of the preset channel transmission difficulty value, the user can determine it according to the actual application scenario. The larger the value of the preset channel transmission difficulty value, the greater the user's demand for multi-segment interval transmission. Provide a method for determining the value of the preset channel transmission difficulty value. Detect the historical records of the user's multi-segment interval transmission, and record the average value of the channel transmission difficulty values corresponding to the historical records that can meet the user's needs as the preset channel transmission difficulty value; Single-segment interval transmission includes: transmitting each data segment in a single data segment set at intervals, and the transmission interval is positively correlated with the channel transmission difficulty value. The transmission interval is the interval time length for transmitting two adjacent data segments; Multi - segment interval transmission includes: evenly dividing each data segment in a single set of data segments into several segment combinations. Each segment combination contains several data segments, and the number of data segments in each segment combination is the same. The number of data segments in a single segment combination is positively correlated with the path evaluation coefficient, and the combined transmission interval is positively correlated with the channel transmission difficulty value. The combined transmission interval is the length of the interval time for transmitting two adjacent segment combinations.

[0041] Specifically, if the delay coefficient is greater than or equal to the preset delay coefficient or the adjustment comparison coefficient is greater than or equal to the preset adjustment comparison coefficient, the optimization method is to determine the priority adjustment method according to the priority pre - adjustment coefficient; If the priority pre - adjustment coefficient is less than the preset priority pre - adjustment coefficient, the priority adjustment method is partial priority replacement; If the priority pre - adjustment coefficient is greater than or equal to the preset priority pre - adjustment coefficient, the priority adjustment method is overall priority replacement.

[0042] Among them, the moment with the shortest time interval from the current moment and at which all corresponding data segments are transmitted to the change analysis node is recorded as the adjacent reference moment. The delay coefficient = the transmission duration corresponding to the adjacent reference moment - the average value of the transmission durations corresponding to each moment in each historical record that can meet the user's requirements. The transmission duration corresponding to a single moment is the time used to transmit the data segments at that moment to the change analysis node; The adjustment comparison coefficient = |the standard deviation of the segment difference degrees corresponding to the set of data segments at the current moment - the standard deviation of the segment difference degrees corresponding to the set of data segments at the adjacent reference moment|; The priority pre - adjustment coefficient = the adjustment comparison coefficient × the pre - delay coefficient; For the values of the preset delay coefficient, the preset adjustment comparison coefficient, and the preset priority pre - adjustment coefficient, users can determine them according to the actual application scenario. The smaller the values of the preset delay coefficient and the preset adjustment comparison coefficient, the greater the user's need to determine the priority adjustment method according to the priority pre - adjustment coefficient. Provide a method. For the values of the preset delay coefficient and the preset adjustment comparison coefficient, detect the historical records of users determining the priority adjustment method according to the priority pre - adjustment coefficient, and record the average value of the delay coefficients corresponding to the historical records that can meet the user's requirements as the preset delay coefficient, and record the average value of the adjustment comparison coefficients corresponding to the historical records that can meet the user's requirements as the preset adjustment comparison coefficient. The larger the value of the preset priority pre - adjustment coefficient, the greater the user's need for partial priority replacement. Provide a method for the value of the preset priority pre - adjustment coefficient. Detect the historical records of users performing partial priority replacement, and record the priority pre - adjustment coefficient corresponding to the historical records that can meet the user's requirements as the preset priority pre - adjustment coefficient; Partial priority replacement includes: detecting the adjustment reference value corresponding to each set of data segments, increasing the transmission priority coefficient corresponding to the set of data segments whose adjustment reference value is greater than the preset adjustment reference value, and the increase value of the transmission priority coefficient corresponding to a single set of data segments has a positive correlation with the adjustment reference value corresponding to this set of data segments; The way to confirm the adjustment reference value is as follows. For a single set of data segments, denote this set of data segments as the target set, denote each set of data segments corresponding to the adjacent reference time as the reference set, and denote the average value of the sub - transmission durations corresponding to the reference sets whose similarity coefficient with the target set is greater than the preset similarity coefficient as the adjustment reference value corresponding to the target set. The sub - transmission duration corresponding to a single reference set is the time used to transmit each data segment in this reference set to the change analysis node. The similarity coefficient between two sets of data segments = 1 - (the absolute value of the difference between the transmission demand coefficients corresponding to the two sets of data segments / the larger value of the transmission demand coefficients corresponding to the two sets of data segments); Overall priority replacement includes: determining the transmission priority coefficient corresponding to each set of data segments according to the adjustment comparison coefficient, and the adjustment comparison coefficient = transmission demand coefficient / adjustment reference value; For the values of the preset adjustment reference value and the preset similarity coefficient, the user can determine them according to the actual application scenario. The greater the user's demand for improving the transmission efficiency of changed data, the smaller the value of the preset adjustment reference value and the larger the value of the preset similarity coefficient. Provide a set of values for the preset adjustment reference value and the preset similarity coefficient. Detect the historical records of the user's partial priority replacement, and denote the average value of the adjustment reference values corresponding to each set of data segments whose transmission priority coefficient is adjusted in the historical records that can meet the user's needs as the preset adjustment reference value, and the preset similarity coefficient is 70%.

[0043] Specifically, if the delay coefficient is less than the preset delay coefficient and the adjustment comparison coefficient is less than the preset adjustment comparison coefficient, the optimization method is to decrease the adjustment of the set transmission interval; The decrease value of the set transmission interval has a negative correlation with the abnormal evaluation coefficient.

[0044] Among them, the abnormal evaluation coefficient = delay coefficient + adjustment comparison coefficient; The set transmission interval is the time interval for transmitting two adjacent sets of data segments. It should be noted that in the present invention, the initial transmission interval has a negative correlation with the total amount of data segment sets.

[0045] So far, the technical solution of the present invention has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0046] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A real-time acquisition method for a database, characterized in that: include: Determine the database status based on the abnormal frequency flow coefficient and the compatible impact coefficient, and determine the tuning setting method based on the database status, such as trigger tuning based on the trigger impact coefficient or cyclic tuning based on the performance load coefficient; Determine the tuning statement category based on the high-frequency execution coefficient and sensitive bottleneck threshold, and perform tuning on the high-frequency impact statements; Determine the data division method of the change log to be analyzed according to the tuning comparison coefficient and the dynamic load intensity to obtain a number of data fragments. The data division method is to divide the data into uniform fragments according to the evaluation deviation index, or to determine the dynamic division method according to the attribute fixity and the range coordination coefficient; The dynamic division method is to perform association division according to attribute similarity and attribute representation value, or to perform multi-dimensional division according to influence similarity and hash similarity; Determine a data segment set according to the distribution adaptability, and determine the distribution mode of each data segment set as multi-path parallel distribution or single-path interval distribution according to the segment difference and the pre-delay coefficient; The optimization method is determined according to the delay coefficient and the adjustment comparison coefficient to adjust the transmission interval or the priority adjustment method is determined according to the priority pre-adjustment coefficient, and the priority adjustment method is partial priority replacement or overall priority replacement.

2. The real-time data collection method for a database according to claim 1, characterized in that: If the database status is that the abnormal frequency flow coefficient is greater than or equal to the preset abnormal frequency flow coefficient or the compatible impact coefficient is greater than or equal to the preset compatible impact coefficient, the tuning setting method is to perform cyclic tuning according to the performance load factor.

3. The real-time acquisition method for a database according to claim 2, characterized in that: If the database status is that the abnormal frequency flow coefficient is less than the preset abnormal frequency flow coefficient and the compatible impact coefficient is less than the preset compatible impact coefficient, the tuning setting method is to perform triggered tuning according to the trigger impact coefficient.

4. The real-time acquisition method for a database according to claim 3, characterized in that: The tuning statement category is determined based on the high-frequency execution coefficient and the sensitive bottleneck threshold. The tuning statement categories include: High-frequency impact statements whose high-frequency execution coefficient is greater than or equal to the preset high-frequency execution coefficient or whose sensitive bottleneck threshold is greater than or equal to the preset sensitive bottleneck threshold; A low-frequency impact statement whose high-frequency execution coefficient is less than a preset high-frequency execution coefficient and whose sensitive bottleneck threshold is less than a preset sensitive bottleneck threshold.

5. The real-time acquisition method for a database according to claim 4, characterized in that: The data division method of the change log to be analyzed is determined based on the tuning comparison coefficient and the dynamic load intensity, including: If the tuning comparison coefficient is less than the preset tuning comparison coefficient and the dynamic load intensity is less than the preset dynamic load intensity, the data division method is to divide the data into uniform segments according to the evaluation deviation index; If the tuning comparison coefficient is greater than or equal to the preset tuning comparison coefficient or the dynamic load intensity is greater than or equal to the preset dynamic load intensity, the data division method is to determine the dynamic division method according to the attribute fixity and the range coordination coefficient.

6. The real-time data collection method for a database according to claim 5, characterized in that: If the attribute fixity is greater than or equal to the preset attribute fixity and the range coordination coefficient is greater than or equal to the preset range coordination coefficient, the dynamic division method is to perform associated division according to the attribute similarity and the attribute representation value.

7. The real-time data collection method for a database according to claim 6, characterized in that: If the attribute fixity is less than the preset attribute fixity or the range coordination coefficient is less than the preset range coordination coefficient, the dynamic division method is to perform multi-dimensional division according to the impact similarity and hash similarity.

8. The real-time data collection method for a database according to claim 7, characterized in that: The distribution method of each data segment set is determined according to the segment difference and the pre-delay coefficient, including: For a single data fragment collection, If the segment difference is greater than or equal to the preset segment difference or the pre-delay coefficient is greater than or equal to the preset pre-delay coefficient, the distribution mode is multi-path parallel distribution; If the segment difference is less than the preset segment difference and the pre-delay coefficient is less than the preset pre-delay coefficient, the segment distribution mode is single-path interval distribution.

9. The real-time data collection method for a database according to claim 8, characterized in that: If the delay coefficient is greater than or equal to the preset delay coefficient or the adjustment comparison coefficient is greater than or equal to the preset adjustment comparison coefficient, the optimization method is to determine the priority adjustment method according to the priority pre-adjustment coefficient; If the priority pre-adjustment coefficient is less than the preset priority pre-adjustment coefficient, the priority adjustment method is partial priority replacement; If the priority pre-adjustment coefficient is greater than or equal to the preset priority pre-adjustment coefficient, the priority adjustment method is overall priority replacement.

10. The real-time data collection method for a database according to claim 9, characterized in that: If the delay coefficient is less than the preset delay coefficient and the adjustment comparison coefficient is less than the preset adjustment comparison coefficient, the optimization method is to reduce the adjustment for the set transmission interval; The reduction value of the collective transmission interval is negatively correlated with the abnormality assessment coefficient.

Images