An artificial intelligence data aggregation method based on big data

Abstract

The present invention relates to the field of data processing technology, specifically to an artificial intelligence data aggregation method based on big data, comprising the following steps: collecting raw traffic monitoring data, removing duplicate data and correcting erroneous data through data cleaning and preprocessing, and generating preliminary cleaned data; applying dynamic index adjustment to the preliminary cleaned data, dynamically adjusting the depth and branches of the index tree according to the access frequency and type of the data item, and obtaining optimized index data. In the present invention, during the dynamic index adjustment process, the index tree structure is dynamically optimized according to the data access frequency and type, thereby improving the retrieval efficiency of high-frequency data and reducing the storage overhead of low-frequency data. Hierarchical data compression processes data in layers, and significantly reduces the use of storage space by combining lightweight compression and advanced compression algorithms, laying the foundation for large-scale data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an artificial intelligence data aggregation method based on big data. Background Art

[0002] The field of data processing technology mainly studies data collection, storage, processing and analysis methods, and is committed to converting massive, multi-source and complex data into structured or semi-structured information. This field covers multiple directions such as data cleaning, data mining, pattern recognition, distributed computing, parallel processing, etc., and is widely used in industries such as industry, finance, transportation, medical care and smart cities. Artificial intelligence data aggregation methods are designed to clean, aggregate and intelligently process complex multi-source monitoring data in the transportation field.

[0003] In the prior art, in terms of data indexing, the index structure design usually adopts a static approach, lacks the ability to dynamically adjust to changes in data access frequency and type, and is difficult to adapt to complex and changing data access requirements, which prolongs the query response time of high-frequency data. In the data storage link, hierarchical management is not implemented according to data characteristics and access distribution, resulting in low storage space utilization, and may also reduce retrieval efficiency under large-scale data storage. Therefore, improvements are needed. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose an artificial intelligence data aggregation method based on big data.

[0005] In order to achieve the above purpose, the present invention adopts the following technical solution, an artificial intelligence data aggregation method based on big data, comprising the following steps:

[0006] Collecting raw traffic monitoring data, removing duplicate data and correcting erroneous data through data cleaning and preprocessing, and generating preliminary cleaned data; applying dynamic index adjustment to the preliminary cleaned data, dynamically adjusting the depth and branches of the index tree according to the access frequency and type of the data items, and obtaining optimized index data;

[0007] Based on the optimized index data, hierarchical data compression technology is implemented to perform hierarchical processing on the data to generate hierarchical compressed data; the hierarchical compressed data is processed in parallel, each data substream is processed on a different computing node, and a parallel processing result is obtained;

[0008] Performing statistical analysis on the parallel processing results, identifying data access patterns using pattern recognition, and generating data access pattern analysis results; adjusting the data storage structure and query algorithm based on the data access pattern analysis results to obtain an optimized data storage structure;

[0009] Based on the optimized data storage structure, data is aggregated, and machine learning models are trained to adjust query processing strategies, optimize query plans and data scanning paths, and generate intelligent query optimization models.

[0010] Preferably, the steps of obtaining the preliminary cleansing data are:

[0011] Through web crawlers and API calls, we collect raw data from traffic monitoring, including temperature, humidity, and vehicle flow, and standardize the data format, including unified timestamp format and data encoding, to obtain a standardized data set.

[0012] Based on the standardized data set, duplicate records are detected and deleted through a hash function, and erroneous data in the data set that does not conform to physical laws is corrected or deleted to obtain a cleaned data set;

[0013] Based on the cleaned data set, interpolation is performed using neighboring data, and feature scaling and missing value processing are performed on the data to obtain preliminary cleaned data.

[0014] Preferably, the step of obtaining the optimized index data is:

[0015] Classify the preliminary cleansed data, count the access frequency of each type of data, stratify the data, each stratum represents a different data type, extract the data type and frequency with the highest number of accesses, and obtain data access frequency records;

[0016] Based on the data access frequency record, the adjustment parameters of the index tree are calculated, including the index tree depth and the number of branches, and the calculation formula is:

[0017]

[0018] and

[0019]

[0020] in, Represents the data access frequency, Represents the number of levels of the data type, is the index tree depth, is the number of branches;

[0021] Based on the calculated index tree depth and branch number, the index tree is structurally adjusted to adjust the index depth and branch number of the node corresponding to each data type to obtain optimized index data.

[0022] Preferably, the step of obtaining the layered compressed data is:

[0023] Analyze the optimized index data, classify the data according to access frequency and update frequency, classify high-frequency data into the first layer, and classify other data into the second layer, to obtain data hierarchical classification;

[0024] Based on the data level classification, the compression rate of each layer is calculated using the following formula:

[0025]

[0026] in, For the The compression ratio of the layer, Representative The access frequency of layer data, Represents the frequency of data modification, is the amount of redundant data, is the level coefficient, is the basic compression efficiency;

[0027] Based on the compression rate, the first layer and the second layer data in the data hierarchy classification are compressed by a compression algorithm to obtain layered compressed data.

[0028] Preferably, the steps of obtaining the parallel processing results are:

[0029] Distributing the layered compressed data to multiple computing nodes, allocating resources according to the compression level and access frequency of each data layer, and obtaining a data distribution result;

[0030] Based on the data distribution result, each computing node executes the data processing task in parallel, processes the data sub-streams by multi-threading, and obtains the node processing result;

[0031] Based on the node processing results, the outputs of each node are summarized, and the processing data of each node are synchronized to obtain the parallel processing results.

[0032] Preferably, the steps of obtaining the data access pattern analysis result are:

[0033] Analyze each data stream in the parallel processing result, extract the timestamp, frequency distribution and access interval of the data access, combine them with the source characteristics of the data stream, construct a multidimensional feature vector, and obtain an access pattern feature set;

[0034] Based on the access pattern feature set, the comprehensive matching score of each pattern is calculated using the following formula:

[0035]

[0036] in, For the The comprehensive matching score of the data access patterns, is the access frequency of the pattern, is the characteristic complexity of the pattern, is the mean time interval of the mode, is the source data importance of the model, is the number of data sub-streams in this mode, is the normalized weight of the pattern;

[0037] Based on the comprehensive matching score, combined with the characteristic distribution and clustering results of each pattern, the data access pattern is identified and classified to obtain a data access pattern analysis result.

[0038] Preferably, the steps of obtaining the optimized data storage structure are:

[0039] Based on the data access pattern analysis result, the distribution of high-frequency access data and low-frequency access data is counted, high-frequency data is defined as hot data, low-frequency data is defined as cold data, and hot data and cold data classification results are generated;

[0040] According to the classification results of the hot data and the cold data, the data storage structure is reorganized, the hot data is stored in a high-speed storage medium, the high-speed storage medium includes a memory or an SSD, and the cold data is stored in a low-speed storage device, the low-speed storage device includes a mechanical hard disk, to form a tiered storage strategy;

[0041] Based on the hierarchical storage strategy, the query algorithm logic is optimized, the query path is adjusted, hot data is retrieved first, and an optimized data storage structure is generated.

[0042] Preferably, the steps of obtaining the intelligent query optimization model are:

[0043] Based on the optimized data storage structure, hot data and cold data in the hierarchical storage are aggregated, hot data is divided into training data according to access frequency and timestamp, cold data is sorted into test data, and training sets and test sets are initialized according to data feature distribution to obtain aggregated training data;

[0044] According to the aggregated training data, a loss function of the machine learning model is constructed, and the expression is:

[0045]

[0046] in, is the loss function value, Indicates The query feature value of samples, Indicates The scan path characteristic value of samples, For query time, For the The query path depth of each data, is the total number of samples;

[0047] Based on the loss function and combined with the updated results of each round of gradient optimization, the query processing strategy is adjusted and the query plan is redesigned to obtain an intelligent query optimization model.

[0048] Compared with the prior art, the advantages and positive effects of the present invention are:

[0049] In the present invention, during the dynamic index adjustment process, the index tree structure is dynamically optimized according to the data access frequency and type, thereby improving the retrieval efficiency of high-frequency data and reducing the storage overhead of low-frequency data. Hierarchical data compression processes data in layers, and greatly reduces the use of storage space by combining lightweight compression and advanced compression algorithms, laying the foundation for large-scale data processing. In the parallel processing link, multiple computing nodes are used to synchronously process data substreams, optimize the data stream distribution, and improve the parallel computing capability and scalability of the system. Based on the pattern recognition of parallel processing results, by identifying complex data access patterns, the access characteristics of different categories of data are accurately summarized, providing a basis for further adjustment of the storage structure. In the storage structure optimization, the data access pattern analysis results are used to redesign the data layout, and the query path logic is adjusted at the same time, shortening the data retrieval time. In the query optimization link, dynamic query strategy adjustment and path optimization are realized by aggregating data and training machine learning models, completing the construction of intelligent query plans in complex traffic monitoring data scenarios, and improving data access efficiency and response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the steps of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] See also Figure 1 The present invention provides a technical solution, an artificial intelligence data aggregation method based on big data, comprising the following steps:

[0053] Collect raw traffic monitoring data, remove duplicate data and correct erroneous data through data cleaning and preprocessing, and generate preliminary cleaned data; apply dynamic index adjustment to the preliminary cleaned data, dynamically adjust the depth and branches of the index tree according to the access frequency and type of data items, and obtain optimized index data;

[0054] Based on the optimized index data, hierarchical data compression technology is implemented to process the data in layers to generate layered compressed data; the layered compressed data is processed in parallel, and each data substream is processed on a different computing node to obtain parallel processing results;

[0055] Perform statistical analysis on the parallel processing results, use pattern recognition to identify data access patterns, and generate data access pattern analysis results; based on the data access pattern analysis results, adjust the data storage structure and query algorithm to obtain an optimized data storage structure;

[0056] Based on the optimized data storage structure, data is aggregated, and machine learning models are trained to adjust query processing strategies, optimize query plans and data scanning paths, and generate intelligent query optimization models.

[0057] The steps to obtain preliminary cleansing data are:

[0058] Through web crawlers and API calls, we collect raw data from traffic monitoring, including temperature, humidity, and vehicle flow, and standardize the data format, including unified timestamp format and data encoding, to obtain a standardized data set.

[0059] Based on the standardized data set, duplicate records are detected and deleted through hash functions, and erroneous data in the data set that does not conform to physical laws is corrected or deleted to obtain a cleaned data set;

[0060] Based on the cleaned data set, interpolation is performed using neighboring data, and feature scaling and missing value processing are performed on the data to obtain preliminary cleaned data.

[0061] Specifically, through web crawlers and API calls, based on the public traffic data source address and related interface parameter descriptions, an automatic script is written to periodically send requests and obtain the returned data packets, parse the temperature, humidity and vehicle flow fields, and judge whether the temperature falls within a reasonable range from 0°C to 90°C during parsing, judge whether the humidity meets the relative humidity specification from 0% to 100%, and evaluate the vehicle flow from 0 to 1000 vehicles / hour. The above ranges are set with reference to the historical statistics of the transportation department and common industry guidelines, and the data structure is matched in JSON format. After extraction, the records are classified according to the timestamp, and the data that is not in the valid range is marked as items that may need to be checked later. The value type of temperature, humidity, and vehicle flow is set to integer or floating point number. After reading, they are merged according to the pre-set time interval to summarize into a relatively complete set of original data. Finally, they are processed according to a unified coding format, the date and time are converted into the form of YYYY-MM-DDHH:MM:SS, and the corresponding relationship between each data is recorded. After all are completed, these data are arranged in the same table to obtain a standardized data set.

[0062] Based on the standardized data set, the field combination used for hashing is first selected, including timestamp and traffic flow information, and the common hash algorithm is used to generate the corresponding hash value to identify each record. When it is detected that the hash values ​​of multiple records are the same and there is no obvious difference in the temperature and humidity fields, they are determined to be duplicate records and deleted. Records with temperatures outside the range of -50℃ to 200℃ are marked, and records with humidity below 0% or above 100% are also marked. If the traffic flow is negative or greater than 10,000, it is regarded as an abnormal item. These boundaries are mainly set with reference to the sensor technical parameters and the historical range of traffic monitoring that has been collected. When there are obvious data errors after comparison, the record will be deleted or corrected. If the approximate value can be inferred from the previous and next time periods, the temperature, humidity or traffic flow of the record will be reset. If it cannot be confirmed, the corresponding mark will be retained for subsequent re-verification. After all records are detected and modified, the processed data is summarized to obtain the cleaned data set.

[0063] Based on the cleaned data set, refer to the two values ​​before and after each record in the time dimension and perform nearest neighbor interpolation. When adjacent data is available and the time interval is within 5 minutes, the average or linear extrapolation value of these two values ​​is used to replace the missing items of the current record. If the interval exceeds 5 minutes, it is considered that interpolation is temporarily unavailable and the mark is retained. Then, the data is feature scaled according to the reasonable range of items such as temperature, humidity, and traffic flow. For example, the temperature has a lower limit of -50℃ and an upper limit of 200℃, and the humidity has a reference interval of 0% to 100%. The traffic flow is set to 0 and 10000 as acceptable boundaries. The difference between the minimum and maximum values ​​of each field is calculated based on the lower and upper limits, and the measured values ​​are mapped to between 0 and 1. If there are still data marked as abnormal or null values ​​in the process, it is compared with the fluctuations in the adjacent time periods to determine whether to abandon them. These fluctuation thresholds are determined by the historical statistical frequency. Finally, the available records are retained and the values ​​are updated in the unified list to obtain preliminary cleaned data.

[0064] The steps to obtain the optimized index data are:

[0065] Classify the preliminary cleansed data, count the access frequency of each type of data, stratify the data, each layer represents a different data type, extract the data type with the highest number of accesses and frequency, and obtain the data access frequency record;

[0066] Based on the data access frequency records, the index tree adjustment parameters are calculated, including the index tree depth and the number of branches. The calculation formula is:

[0067]

[0068] and

[0069]

[0070] in, Represents the data access frequency, Represents the number of levels of the data type, is the index tree depth, is the number of branches;

[0071] Based on the calculated index tree depth and branch number, the index tree is structurally adjusted to adjust the index depth and branch number of the node corresponding to each data type to obtain optimized index data.

[0072] Specifically, the preliminary cleaned data is classified, and the temperature, humidity, traffic flow field and other information of each record obtained previously are referred to. They are read one by one and grouped according to the data source identifier and data field type. The number of times the same data type appears in different time ranges is recorded and compared. In the classification process, the temperature data can be compared with the 0℃ to 90℃ interval to check its rationality, the humidity data can be compared with the 0% to 100% interval, and the traffic flow can be compared with the 0 to 1000 vehicles / hour interval. After all the records are scanned and arranged in a table in turn, the frequency of occurrence of each type of data is calculated, and the statistical results are mapped with the category range defined in the early stage to confirm the high-frequency and low-frequency data types. In this process, the types that are accessed more than 600 times and the types that are accessed less than 10 times will be marked. These 600 and The threshold of 10 times is based on the range obtained by the centralized statistics of the number of visits within a week. After obtaining these marked data types, a hierarchical structure is constructed based on the type and number of occurrences of each record. For example, temperature data is divided into one layer, humidity data is divided into another layer, and traffic flow data is divided into the third layer. It is recorded that the temperature type with extremely high access frequency appeared 720 times in the most recent monitoring period, the humidity type appeared 500 times, and the traffic flow appeared 630 times. Then the access information of these data types is sorted and regrouped for the second time, and the chronological order is maintained to check whether there are access requests that exceed the range or deviate significantly from the reference interval. If it is confirmed that extreme access volume has indeed appeared multiple times, the classification status is retained. At this time, after completing the access statistics of each data type, the data access frequency record is obtained.

[0073] The formula is useful because it introduces the frequency of data access and the number of data type levels , can dynamically adjust the depth and number of branches of the index tree in the same index structure to adapt to data types of different levels and different access volumes;

[0074] The parameter acquisition step is to accumulate the number of visits per unit time for each data type through the statistical step obtained above, and form daily, weekly or monthly comprehensive statistical data. After grouping these data, the integrated number of visits is selected in this example. , which is obtained by accumulating the number of request records of each type every hour in the past seven days, and excluding the abnormal fluctuation range;

[0075] The parameter acquisition step is to check all current data types and detect the number of layers they belong to. In this example, the three main types of temperature, humidity, and traffic flow belong to different layers, so the number of layers counted is , this value is obtained by combining and confirming the three categories previously divided;

[0076] Calculation process:

[0077] The first step is to calculate the index tree depth :

[0078]

[0079] Will and Bring in,

[0080]

[0081]

[0082]

[0083] The second step is to calculate the number of branches :

[0084]

[0085] Will Bring in,

[0086]

[0087]

[0088]

[0089] The results show that according to the access times and the number of levels obtained previously, the index tree depth can be set to about 38 or 39 levels and the number of branches can be determined to be 9. By increasing or decreasing the number of nodes, the hierarchical configuration of the index structure can be updated accordingly, and the branches and depth of the index tree can be kept within a suitable range.

[0090] Based on the calculated index tree depth and number of branches, first, according to the previously obtained and These two parameters arrange the structure of the nodes corresponding to each data type, and record the temperature data nodes starting from the first-level tree root, followed by the humidity data nodes, and then expanding layer by layer in combination with the traffic data nodes. When adjusting, we will first confirm whether the actual depth of each node meets the constraint of not exceeding 40 levels. This value comes from the previous operation and maintenance statistics and hardware performance tests, and split the trunk when the number of branch nodes is greater than 9. This split threshold is based on the numerical range summarized after monitoring the storage query time. When the number of branch nodes exceeds 15 at one time, it is recorded as an item that needs to be reviewed. Once it is confirmed that it has exceeded 9 and continues to grow, the structure of the position will be re-split. At the same time, the depth and branch number rules will be repeatedly applied to the temperature data nodes for verification. If it is found that the actual access volume has increased sharply and increased If the statistical results are obtained, new The depth and branches of the index tree are calculated again. When all classification nodes finally meet the control standards of depth and branching, the adjusted hierarchical structure is reconstructed, and the corresponding data types are verified with the index nodes. After confirmation, the optimized index data is obtained.

[0091] The steps to obtain hierarchical compressed data are:

[0092] Analyze the optimized index data and classify the data according to access frequency and update frequency. High-frequency data is classified into the first layer and other data is classified into the second layer to obtain data hierarchical classification.

[0093] Based on the data level classification, the compression rate of each layer is calculated using the following formula:

[0094]

[0095] in, For the The compression ratio of the layer, Representative The access frequency of layer data, Represents the frequency of data modification, is the amount of redundant data, is the level coefficient, is the basic compression efficiency;

[0096] Based on the compression rate, the first layer and the second layer data in the data hierarchy classification are compressed by a compression algorithm to obtain layered compressed data.

[0097] Specifically, the optimized index data is analyzed. After obtaining the index data, the access fields with the mark are read one by one and the access frequency and update frequency are checked. For example, when the access frequency of the temperature category is greater than 500 times and the update frequency is greater than 300 times, it is recorded as a high-frequency category. If the access frequency is between 20 and 100 times but the update frequency is less than 50 times, it is classified as a normal category. The ranges of 500 and 20 times are both derived from the daily statistics of the access and modification volume in the previous week, and are calculated in combination with the average and range of such data. When it is confirmed that it is indeed significantly higher than the average access level or much lower than the regular update frequency, the high-frequency category is recorded. When a new situation occurs, it is included in different stratification ranges, and then the data entries corresponding to each category are summarized and the temperature, humidity, traffic flow and other fields are checked to see if they fall within their respective valid ranges. For example, the temperature is between 0°C and 90°C, the humidity is between 0% and 100%, and the traffic flow is between 0 and 1,000 vehicles per hour. If there are entries that are higher than 10% of the total number of visits for several consecutive days, they are counted as the first layer, and other entries that do not trigger the above conditions are classified as the second layer. The above steps all determine whether they belong to the first or second layer by comparing the previously recorded data. After confirming the hierarchical division of all entries and checking the correspondence with the existing index information, the data hierarchical classification is obtained.

[0098] The formula is useful in that it combines the access frequencies , Modify frequency , Redundant data volume , level coefficient And basic compression efficiency To calculate the compression ratio , so that data at different levels can be adjusted differently according to their respective characteristics during compression processing;

[0099] The steps for obtaining the parameters are as follows: first, extract the number of data accesses of the layer from the previously obtained classification data. For example, when counting a certain high-frequency layer, the number of accesses is accumulated to 2300 times. This value is obtained by monitoring and recording the frequency of each data request for 24 hours for five consecutive days. After merging repeated accesses, this statistical value is kept for subsequent calculations.

[0100] The parameter acquisition step is to record the number of data modifications of the layer. In this example, it is statistically found that a frequently updated data layer has performed 160 overwrite write operations. This number is obtained by parsing the database transaction log and counting key fields such as the operation type "update" or "change";

[0101] The steps to obtain the parameters are as follows: quantifying the amount of redundant data requires combining the deduplication analysis of the data layer. In this example, the duplication rate is about 12%. When deduplication is performed on the total number of records of 14,000 records in this layer, the number of redundant records is 1,680, that is, , which is the sum of the data after comparing the hash of each record;

[0102] The steps for obtaining the parameters are to refer to whether the layer belongs to the first layer or the second layer, and combine the above hierarchical division. In this example For the first layer, this is selected after multiple comparisons of the parameters based on historical compression test results and the timeliness of the data involved in this layer;

[0103] The parameter acquisition step is to select the value based on the existing compression program efficiency index, and after compressing similar scale data multiple times and recording the occupied resources, select ,At this time, the resource consumption and compression effect of the compression program when processing more than 10,000 records are relatively balanced;

[0104] Calculation process:

[0105] The first step is to bring in sample data and let , , , ,

[0106] The second step is to calculate the numerator and denominator:

[0107] Denominator

[0108] molecular

[0109] The third step is to divide the numerator by the denominator:

[0110]

[0111] Step 4: Exponentiation and multiplication :

[0112]

[0113] Multiply by :

[0114]

[0115] The result shows that in this example, the compression rate of the first layer of data is about 1.1118. Its value greater than 1 means that the compression efficiency is not enough to reduce the overall data volume under the calculation logic, and the parameters need to be adjusted. or When the value is less than 1, it usually indicates that the data storage occupancy can be effectively reduced. If it exceeds 1, it may indicate that the tiered compression strategy needs to be adjusted accordingly, which is related to the subsequent compression results of the actual data.

[0116] Based on the compression ratio, first refer to the calculated The first layer and the second layer entries are compared with the same values, and the data of the first layer are mostly concentrated in the statistical interval where the access frequency exceeds 600 times and the modification frequency exceeds 100 times. At the same time, it is confirmed that the amount of redundant data remains between 500 and 3000. This interval comes from the comparative analysis results of the previous monitoring cycle. After confirming the frequency and redundancy of these entries, they are batch-entered into a set of processes specifically used for compression. The specific process includes performing a repeatability scan on each record and sorting it according to the access time, performing differential compression or deletion operations on the part with a repeatability exceeding 20%. The judgment of this 20% refers to the data characteristics of each group of records and compares them. The data accumulated in the previous few weeks are used to determine the threshold, and then the data with more than 200 modifications or more than 600 modifications are checked. If they appear repeatedly in the same time window, they are marked as high-frequency change items and another compression method is used to remove the redundant parts. For the second layer of data, the update frequency is lower and the access volume is between 200 and 600 times. The range also comes from the previously counted access fluctuations. After all the data is processed according to differential compression or batch compression, the difference in byte occupancy before and after compression is compared one by one, and the data information recorded in the compressed output is registered. Finally, the two layers of data are merged to obtain layered compressed data.

[0117] The steps to obtain the parallel processing results are:

[0118] Distribute the layered compressed data to multiple computing nodes, allocate resources according to the compression level and access frequency of each data layer, and obtain data distribution results;

[0119] Based on the data distribution results, each computing node executes data processing tasks in parallel, uses multi-threading to process data sub-streams, and obtains node processing results;

[0120] Based on the node processing results, the output of each node is summarized, the processing data of each node is synchronized, and the parallel processing results are obtained.

[0121] Specifically, the layered compressed data is distributed to multiple computing nodes. First, the access frequency and compression level records obtained previously are referred to, and compared with the pre-established range, for example, the number of accesses between 500 and 2000 is classified as medium demand, and the number of accesses exceeding 2000 is classified as high demand. These values ​​are obtained by the previous analysis and statistics of network traffic and request volume. When distributing, the high-level compressed data is synchronously compared with the access frequency according to the compression level. If it is found that the access volume of the layer of data is more than 10% of the daily average, it is marked as high-demand data, and it is preferentially distributed to computing nodes with higher CPU resources or greater concurrency capabilities. The access frequency between 1% and 10% of the daily average is classified as medium demand and processed by medium-performance nodes. Other data with a frequency lower than 1% or a lower compression level are distributed to basic performance nodes. During the distribution process, it is also determined whether the CPU utilization rate reaches the empirical threshold of 80% based on the current workload of the node. This threshold is summarized by measuring the response speed and stability of the node under different loads for many times. If the CPU utilization rate exceeds 80%, the subsequent data is preferentially distributed to the next idle node. In this way, all the layered compressed data are distributed to obtain the data distribution result.

[0122] Based on the data allocation results, when each computing node executes data processing tasks in parallel, it first reads the allocated data entries and checks their compression level and access times, and then decides whether to enable multi-threading to accelerate processing based on the node configuration. If the size of a single record exceeds 10MB or the inflow frequency exceeds 50 per second, an additional thread is created in the node to split and process the data sub-streams in parallel. Here, 10MB and 50 per second are thresholds obtained by multiple tests based on node bandwidth and I / O rate. Then, each piece of data is compared with its own format one by one and quickly screened according to the effective range of fields such as temperature, humidity, and vehicle flow. For example, the temperature is referenced to 0℃ to 90℃, the humidity is based on 0% to 100% as the benchmark, and the vehicle flow is based on 0 to 1000 vehicles / hour as the judgment interval. If any field is not in this range, it is marked in the abnormal record list inside the node, and the records that meet the range are further processed. After processing, the results are written to the node's temporary storage area and the processing progress is updated. When all records are split and read, the node integrates the processing of all sub-streams and outputs the node processing results.

[0123] Based on the node processing results, we first search for the output of each node, read the classification label and timestamp corresponding to each record, and arrange the data produced by all nodes in a table by time period. In order to avoid concurrent conflicts or data mixing, we compare the consistency of timestamps when synchronizing the processed data of each node, and continue to perform secondary reviews at the field value level according to the previously established valid range. After confirming that it does not exceed the pre-defined temperature, humidity and traffic flow range, we merge the records one by one into a unified data list, and generate a traceable index to identify its source node and corresponding sub-stream. If duplicate records are found in multiple nodes in the same time period, the duplicate items are marked. The duplication judgment here combines the hash value of the record with the field matching degree for judgment. When the number of duplicate items exceeds 2% of the total amount of the entire data segment, the original processing flow of each node will be traced back again to confirm whether there are abnormalities in the allocation or splitting links. Finally, after all node outputs are synchronized, all data information and marking status are integrated to obtain parallel processing results.

[0124] The steps to obtain the data access pattern analysis results are:

[0125] Analyze each data stream in the parallel processing results, extract the timestamp, frequency distribution and access interval of data access, combine them with the source characteristics of the data stream, construct a multi-dimensional feature vector, and obtain the access pattern feature set;

[0126] Based on the access pattern feature set, the comprehensive matching score of each pattern is calculated using the following formula:

[0127]

[0128] in, For the The comprehensive matching score of the data access patterns, is the access frequency of the pattern, is the characteristic complexity of the pattern, is the time interval mean of the mode, is the source data importance of the model, is the number of data sub-streams in this mode, is the normalized weight of the pattern;

[0129] Based on the comprehensive matching score, combined with the characteristic distribution and clustering results of each pattern, the data access patterns are identified and classified to obtain the data access pattern analysis results.

[0130] Specifically, each data stream in the parallel processing results is analyzed. First, the timestamp, access frequency and access interval of each data stream are read one by one from the node output data obtained previously. These values ​​are recorded and compared to see if they fall within the previously set reference range. For example, the time interval is compared with the interval of 5 seconds to 300 seconds to determine the concentration of access, and the access frequency is compared with 0 to 50 times per minute to determine whether it belongs to a high-frequency stream. The source characteristics are recorded by the specific identifiers given during collection. After matching these source identifiers with known temperature, humidity, traffic flow or other categories of fields, a column of tags is generated to distinguish the collection location or collection device corresponding to each data stream. Then, the timestamp, access interval, source characteristics and frequency distribution are recorded. The elements are combined to form a four-dimensional or multi-dimensional vector. If there are missing fields or interval spans exceeding 24 hours in some data streams, the previous records are compared again to confirm whether it is caused by network jitter or device offline. If it is confirmed to be offline data, it is marked separately for subsequent aggregation. When the above access frequency, time interval and source characteristics are compared and added to the vector, these vectors are mapped to a feature matrix in the order of calculation. In the matrix, vectors with the same source or similar time distribution are divided into a set. If the frequency in the set is too high or the interval is too dense, a concentrated access label is affixed. If the access in the set is irregular and the time span is large, it is classified as a scattered feature. Finally, the multi-dimensional vectors corresponding to all data streams are merged to obtain an access pattern feature set.

[0131] The formula is useful because it incorporates the frequency of visits , Feature complexity , mean time interval , Source Data Importance , Number of data sub-flows and the normalized weights of the modes Comprehensive calculations can reflect the matching degree of each data access mode in multiple indicators;

[0132] The step of obtaining the parameters is to record the access frequency of the current mode. For example, a total of 360 accesses were detected in the statistical period within the past 24 hours. These 360 ​​accesses are accumulated from the query volume per hour and exclude abnormal offline time. ;

[0133] The parameter acquisition step is to quantify the feature complexity of the pattern. If the pattern involves three fields, temperature, humidity, and traffic flow, and also adds some data from other sensors, the number of features is large. The number of features can be counted and combined with their standardization coefficients. For example, here the number of features is determined to be 12 and a comprehensive complexity of 1.5 is obtained. Because the more features, the more complex the processing. ;

[0134] The steps to obtain the parameters are to calculate the average interval for the access time distribution of the mode, first filter the timestamps belonging to the same mode from the records, and calculate the time difference of adjacent accesses one by one, and get 320 valid interval values ​​within 24 hours. Sum them and divide them by 320 to get an average interval of about 4.5 minutes, and finally convert them into seconds to 270 seconds, so ;

[0135] The steps for obtaining the parameters are as follows: the importance of the data involved in this mode needs to be cumulatively scored in combination with the source characteristics. For example, this mode involves both real-time road monitoring and temperature records in tunnels. The weight value is obtained by querying the established classification table and combining it with the reference score to calculate 13.2 points, which is used as the source importance. ;

[0136] The step of obtaining the parameters is to count the number of data sub-flows contained in the mode. After analyzing the output of all nodes, it is found that there are 4 sub-flows belonging to the same access mode. These 4 sub-flows are taken as The value of ,This value is obtained by matching the identifier of each sub-flow and its pattern label, and counting is performed after comparing the sub-flow ID and the source address;

[0137] The parameter acquisition step is that the mode standardization weight can be set by comparing the overall contribution of different modes under the same conditions. For example, after looking back at the historical data within three weeks, it is found that this mode accounts for 8% of the total visits and has a certain interaction with other modes. ;

[0138] Calculation process:

[0139] The first step is to substitute , , , , ,

[0140] The second step is to calculate :

[0141]

[0142]

[0143] The third step is to square this value:

[0144]

[0145] Step 4: Calculate :

[0146]

[0147]

[0148] Step 5: Combine

[0149]

[0150]

[0151] The results show that the comprehensive matching score of the pattern is about 2.478. When the value is greater than 2, it usually means that the pattern has a higher access frequency and more complex features. When it is less than 1, it means that the feature distribution of the pattern is not prominent or the frequency is low. The overall score can be used as a threshold reference for subsequent pattern classification.

[0152] Based on the comprehensive matching score, firstly sort the scores of all patterns by size and record the feature complexity and source data importance associated with them. Then, the patterns with scores above 2.0 are classified as high-attention patterns, those below 2.0 but above 1.0 are classified as medium-attention patterns, and those below 1.0 are classified as low-attention patterns. These critical points of 2.0 and 1.0 are obtained by dividing the distribution of matching degrees in historical data statistics. Then, additional classification is performed within each group based on the access time distribution and the number of substreams. For example, patterns that contain both temperature and humidity and have more than 3 substreams are marked as extended patterns, while those that only contain a single field access are marked as single patterns. At this time, the marking of all patterns is combined with the clustering results. If the clustering shows that several patterns are in the same cluster and their scores are all in the high-attention interval, they are considered to have the same type or high similarity features, thus forming the final multi-grouping structure. After this structure is completed, all classifications and their corresponding scores are listed to obtain the data access pattern analysis results.

[0153] The steps to obtain the optimized data storage structure are:

[0154] Based on the data access pattern analysis results, the distribution of high-frequency access data and low-frequency access data is counted, high-frequency data is defined as hot data, low-frequency data is defined as cold data, and hot data and cold data classification results are generated;

[0155] According to the classification results of hot data and cold data, the data storage structure is reorganized to store hot data in high-speed storage media, such as memory or SSD, and cold data in low-speed storage devices, such as mechanical hard disks, to form a tiered storage strategy;

[0156] Based on the tiered storage strategy, the query algorithm logic is optimized, the query path is adjusted, hot data is retrieved first, and an optimized data storage structure is generated.

[0157] Specifically, based on the results of data access pattern analysis, read and organize the previously recorded data access frequencies and access time periods, compare the access distribution of fields such as temperature, humidity, and traffic volume in the past week one by one, and compare each record with the pre-established access frequency range. For example, if the access frequency is greater than 500 times, it is considered to be in the high-frequency range, and if it is less than 50 times, it is considered to be in the low-frequency range. These values ​​are mainly derived from historical trend statistics and multi-day monitoring results. If the access frequency of some records is between 50 and 500 times, it is temporarily regarded as the middle segment and continues to observe whether it meets the high-frequency or low-frequency standards. At the same time, check the time interval of the records to determine whether there is continuous high-density access. When the number of accesses for multiple consecutive hours is within the statistical period, When the proportion exceeds 10%, it is marked as a possible hotspot area. The 10% threshold is obtained by comparing the total visits and the peak distribution. If a data type has a very low visit volume over a long period of time and only appears once every 24 hours or several days, it is classified as unpopular. After merging these marking results, the distribution chart is regenerated and invalid data is proofread. When it is confirmed that the above interval divisions and markings meet the previously defined thresholds, the data type corresponding to the high-frequency interval is named hot data, and the data corresponding to the low-frequency interval is named cold data. At the same time, the differences between these hot data and cold data in terms of source, number of fields and access path are recorded. Finally, all entries that meet the definition are classified to generate hot data and cold data classification results.

[0158] According to the classification results of hot data and cold data, we first extract high-frequency access records one by one and check their size and time distribution. When the size of a single record exceeds 10MB or the single access frequency reaches more than 20 times within one minute, this type of entry is marked as urgently in need of accelerated processing. These size and frequency standards are obtained by summarizing and averaging the actual stress tests of multiple projects. After confirming this type of high-frequency record, this type of record is arranged to be stored in memory or SSD through the previously summarized category information. The selection range of SSD is mainly on devices that can support continuous write speeds of more than 300MB / s and can carry high IOPS. The remaining low-frequency records with only dozens of access times or no more than 5 times a day are classified as cold data. For these cold data, if the total amount and the size of a single entry are maintained between hundreds of KB and several MB and the access cycle often exceeds several hours, mechanical hard disks are deployed for storage. When all hot data and cold data are recorded on the storage medium, a corresponding relationship table is generated inside the system to represent the entries and access levels contained in each storage medium, and finally a hierarchical storage strategy is formed.

[0159] Based on the tiered storage strategy, we first adjust the query logic according to the memory or SSD location where the high-frequency data is located, and count the common query keywords and their corresponding hot data locations. If the keyword has been requested more than 100 times in the past week and the main access field involves temperature or humidity, its path will be prioritized to the index where the memory or SSD is located. This 100 times is an empirical threshold established after comprehensive multi-day access counts and low latency requirements. At the same time, the low-frequency data path is redirected to the mechanical hard disk storage area and the index level is marked as a cold path. If the retrieval request focuses on traffic flow but the number of accesses is only once a day or less, it is automatically matched to the cold storage path. When executing the query, the request conditions entered by the user are compared. If it meets the high-frequency data characteristics, the memory or SSD resources are directly scheduled to complete the retrieval and return the results. If it matches the low-frequency data characteristics, it jumps to the mechanical hard disk for reading. During the whole process, the system will record the query time and node response status so that these query routes can be fine-tuned again later. When all queries are completed through the dynamic path, the optimized data storage structure at the current moment is generated in combination with the final query records and response status.

[0160] The steps to obtain the intelligent query optimization model are:

[0161] Based on the optimized data storage structure, hot data and cold data in the hierarchical storage are aggregated, hot data is divided into training data according to access frequency and timestamp, cold data is sorted into test data, and training and test sets are initialized according to data feature distribution to obtain aggregated training data;

[0162] Based on the aggregated training data, the loss function of the machine learning model is constructed, and the expression is:

[0163]

[0164] in, is the loss function value, Indicates The query feature value of samples, Indicates The scan path characteristic value of samples, For query time, For the The query path depth of each data, is the total number of samples;

[0165] Based on the loss function and combined with the updated results of each round of gradient optimization, the query processing strategy is adjusted and the query plan is redesigned to obtain an intelligent query optimization model.

[0166] Specifically, based on the optimized data storage structure, we first select the actual storage locations of hot data and cold data in the hierarchical storage, read the access frequency and timestamp distribution of hot data respectively and summarize them, and classify the records with access times exceeding 10% of the daily total into the high-frequency set. This 10% ratio is the empirical threshold calculated by accumulating all access times in the past seven days according to the daily average. If a single data entry appears repeatedly in multiple adjacent time periods and the timestamp difference is less than 10 minutes, it is regarded as continuously active. This 10 minutes is determined in combination with the request peak interval monitored in the early stage. After confirming that these active records are all within the selected high-frequency range, they are identified as the main source of training data and spliced ​​together in chronological order. In the process of summarizing, when classifying records that appear only once in more than 24 hours or have a total of less than 10 access times in a week, check their access times in the adjacent time periods. Whether it also maintains an extremely low frequency. If it is indeed in a low usage state, it is included in the cold data range. These values ​​below 10 times are based on the global sorting of the access data and combined with the reference obtained by the last 10% quantile. Then the cold data is reintegrated, the field information and source location are marked, and it is organized into a preliminary list of test data. When constructing the training set and the test set, it is also checked whether the temperature, humidity, traffic flow and other characteristics in each record meet the valid range established before. For example, the temperature is compared with the range of 0℃ to 90℃, the humidity is compared with 0% to 100%, and the traffic flow is compared with 0 to 1000 vehicles / hour. If all of them meet the requirements, the item will be formally included in the training or test sequence. If not, it will be marked as abnormal data for subsequent investigation. Finally, the sorting results of the hot data and the cold data are merged to obtain the overall training data and test data framework. After the parameter inventory and time sorting are correct, the aggregated training data is obtained.

[0167] The formula is beneficial in that it combines the query feature values ​​of each sample , scanning path characteristic value , query time , query path depth And the total number of samples ,The comprehensive deviation of the query process on time and path is measured through the balance between the numerator and the denominator;

[0168] The parameter acquisition step is to count the query keywords and access fields of each record from the training data obtained above, and quantify them. For example, high-frequency keywords are assigned a value range of 5 to 10, medium-frequency keywords are assigned a value range of 2 to 4, and low-frequency keywords are assigned a value of 1. If a record contains multiple keywords, they are added together to form a total feature value. In this example, , wait;

[0169] The steps to obtain the parameters are as follows: , which represents the scan path feature value. The path that accesses the memory or SSD needs to be recorded as a smaller value, such as 1 or 2, indicating a faster access speed. The slower media such as mechanical hard disks are assigned a value of 3 or 4. In this example, if the first record is located in the SSD, then If the second record is located on a mechanical hard disk, then ;

[0170] The steps for obtaining the parameters are: query time, which records the time taken from initiating a query to obtaining the result, in milliseconds or seconds. In this example, after quantizing all training samples, the query time for the first record is about 120ms, and the query time for the second record is about 350ms. , ;

[0171] The parameter acquisition step is that the query path depth of each data can be determined by referring to the level of the index tree or hierarchical storage. If the data is in a shallow layer, the depth is smaller and is recorded as 1 or 2. If the data is in a deeper layer, it is recorded as 3 or 4. In this example, the depth of the first record path is recorded as 1, and the depth of the second record path is recorded as 2. After sorting, it is formed , ;

[0172] The parameters are obtained in the following steps: the total number of samples comes from the number of available items in the training data after the previous aggregation. In this example, Give a demonstration;

[0173] Calculation process:

[0174] The first step is to bring in data:

[0175] , , ,

[0176] , , ,

[0177] The second step is to calculate the error term for the first record:

[0178]

[0179]

[0180]

[0181] The square of this value is approximately 6509.86

[0182] The third step is to calculate the error term of the second record:

[0183]

[0184]

[0185]

[0186] The square of this value is approximately 23260.80

[0187] Step 4: Add the sum and take the square root:

[0188]

[0189]

[0190] The results show that the loss function value is higher under the example parameter setting, which shows that there is a significant difference between the query features and the actual query time. When it is less than 50, it usually indicates that the query speed is relatively matched, and when it is greater than 100, it indicates a large deviation. This result can be used in the gradient optimization process to iteratively adjust the path or index strategy.

[0191] Based on the loss function, the parameter update list obtained from each round of previous training is retrieved first, and the change curve of the current query time under different paths is summarized. Whether there is a situation where the query speed is significantly slower than the historical average is recorded and marked. When the number of marks exceeds 5% of all data, the 5% value is a relatively stable threshold determined after multiple sampling statistics, that is, if 5% of consecutive samples show obvious deviations, the configuration needs to be further checked. Then, the query processing strategy is readjusted according to the difference between the query feature value and the scan path feature value. The data with higher depth but high frequency of use are re-labeled and included in the training set, and the query time parameters and the updated scan path are put into a new round of training process. At this time, in each iteration, the new path depth correction is extracted from the gradient optimization results of the previous round and applied to the path selection logic. At the same time, the items with too long query time or very low frequency are sorted as control items to observe the subsequent convergence degree. After multiple rounds of iterations, when the cumulative query time tends to be stable and the difference no longer fluctuates greatly, the training process is finally finalized and a relatively complete query plan is formed, which is combined with the previously updated path information to obtain an intelligent query optimization model.

[0192] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. An artificial intelligence data aggregation method based on big data, characterized in that: The following steps are involved: Collect raw traffic monitoring data, remove duplicate data and correct erroneous data through data cleaning and preprocessing, and generate preliminary cleaned data; Applying dynamic index adjustment to the preliminary cleansed data, dynamically adjusting the depth and branches of the index tree according to the access frequency and type of the data items, to obtain optimized index data; Based on the optimized index data, a hierarchical data compression technology is implemented to process the data in layers to generate layered compressed data; Processing the layered compressed data in parallel, with each data substream being processed on a different computing node, to obtain a parallel processing result; Performing statistical analysis on the parallel processing results, identifying data access patterns using pattern recognition, and generating data access pattern analysis results; Based on the data access pattern analysis results, adjusting the data storage structure and query algorithm to obtain an optimized data storage structure; Based on the optimized data storage structure, aggregate data, train machine learning models, adjust query processing strategies, optimize query plans and data scanning paths, and generate intelligent query optimization models; The steps for obtaining the layered compressed data are: Analyze the optimized index data, classify the data according to access frequency and update frequency, classify high-frequency data into the first layer, and classify other data into the second layer, to obtain data hierarchical classification; Based on the data level classification, the compression rate of each layer is calculated using the following formula: in, For the The compression ratio of the layer, Representative The access frequency of layer data, Represents the frequency of data modification, is the amount of redundant data, is the level coefficient, is the basic compression efficiency; Based on the compression rate, compressing the first layer and the second layer of data in the data hierarchical classification by a compression algorithm to obtain hierarchical compressed data; The steps for obtaining the data access pattern analysis result are: Analyze each data stream in the parallel processing result, extract the timestamp, frequency distribution and access interval of the data access, combine them with the source characteristics of the data stream, construct a multidimensional feature vector, and obtain an access pattern feature set; Based on the access pattern feature set, the comprehensive matching score of each pattern is calculated using the following formula: in, For the The comprehensive matching score of the data access patterns, is the access frequency of the pattern, is the characteristic complexity of the pattern, is the mean time interval of the mode, is the source data importance of the model, is the number of data sub-streams in this mode, is the normalized weight of the pattern; Based on the comprehensive matching score, combined with the characteristic distribution and clustering results of each pattern, the data access pattern is identified and classified to obtain a data access pattern analysis result.

2. The artificial intelligence data aggregation method based on big data according to claim 1 is characterized in that: The steps for obtaining the preliminary cleaning data are: Through web crawlers and API calls, we collect raw data from traffic monitoring, including temperature, humidity, and vehicle flow, and standardize the data format, including unified timestamp format and data encoding, to obtain a standardized data set. Based on the standardized data set, duplicate records are detected and deleted through a hash function, and erroneous data in the data set that does not conform to physical laws is corrected or deleted to obtain a cleaned data set; Based on the cleaned data set, interpolation is performed using neighboring data, and feature scaling and missing value processing are performed on the data to obtain preliminary cleaned data.

3. The artificial intelligence data aggregation method based on big data according to claim 1 is characterized in that: The steps for obtaining the optimized index data are as follows: Classify the preliminary cleansed data, count the access frequency of each type of data, stratify the data, each stratum represents a different data type, extract the data type and frequency with the highest number of accesses, and obtain data access frequency records; Based on the data access frequency record, the adjustment parameters of the index tree are calculated, including the index tree depth and the number of branches, and the calculation formula is: and ,in, Represents the data access frequency, Represents the number of levels of the data type, is the index tree depth, is the number of branches; Based on the calculated index tree depth and branch number, the index tree is structurally adjusted to adjust the index depth and branch number of the node corresponding to each data type to obtain optimized index data.

4. The artificial intelligence data aggregation method based on big data according to claim 1 is characterized in that: The steps for obtaining the parallel processing results are: Distributing the layered compressed data to multiple computing nodes, allocating resources according to the compression level and access frequency of each data layer, and obtaining a data distribution result; Based on the data distribution result, each computing node executes the data processing task in parallel, processes the data sub-streams by multi-threading, and obtains the node processing result; Based on the node processing results, the outputs of each node are summarized, and the processing data of each node are synchronized to obtain the parallel processing results.

5. The artificial intelligence data aggregation method based on big data according to claim 1 is characterized in that: The steps for obtaining the optimized data storage structure are: Based on the data access pattern analysis result, the distribution of high-frequency access data and low-frequency access data is counted, high-frequency data is defined as hot data, low-frequency data is defined as cold data, and hot data and cold data classification results are generated; According to the classification results of the hot data and the cold data, the data storage structure is reorganized, the hot data is stored in a high-speed storage medium, the high-speed storage medium includes a memory or an SSD, and the cold data is stored in a low-speed storage device, the low-speed storage device includes a mechanical hard disk, to form a tiered storage strategy; Based on the hierarchical storage strategy, the query algorithm logic is optimized, the query path is adjusted, hot data is retrieved first, and an optimized data storage structure is generated.

6. The method for artificial intelligence data aggregation based on big data according to claim 1, characterized in that: The steps for obtaining the intelligent query optimization model are: Based on the optimized data storage structure, hot data and cold data in the hierarchical storage are aggregated, hot data is divided into training data according to access frequency and timestamp, cold data is sorted into test data, and training sets and test sets are initialized according to data feature distribution to obtain aggregated training data; According to the aggregated training data, a loss function of the machine learning model is constructed, and the expression is: ,in, is the loss function value, Indicates The query feature value of samples, Indicates The scan path characteristic value of samples, For query time, For the The query path depth of each data, is the total number of samples; Based on the loss function and combined with the updated results of each round of gradient optimization, the query processing strategy is adjusted and the query plan is redesigned to obtain an intelligent query optimization model.

Images