Artificial intelligence big data processing method and system for smart traffic

Abstract

The invention relates to the technical field of intelligent data analysis, in particular to an artificial intelligence big data processing method and system for intelligent traffic, and the method comprises the following steps: collecting real-time traffic data, removing wrong or incomplete records, classifying and marking frequently accessed data, and generating cleaning and classification results. According to the method, data are collected and cleaned in real time, errors or incomplete records are removed, and the data accuracy is ensured. And then the data is intelligently classified, frequent and low-frequency access data are distinguished, and the data access efficiency is improved. And then analyzing the data by using machine learning, and constructing an access frequency model to provide a scientific basis for optimization of a data storage position. Besides, by dynamically adjusting the positions of the data among different storage levels, the distribution of storage resources is optimized, the overall efficiency of the data processing system is improved, the operation cost is reduced, and the practicability is high.

Description

Technical Field

[0001] The present invention relates to the field of intelligent data analysis technology, and in particular to an artificial intelligence big data processing method and system for smart transportation. Background Art

[0002] The field of data analysis technology is a broad range of methods and processes used to examine, clean, transform, and model data to discover useful information, draw conclusions, and support decision-making. In smart transportation systems, data analysis technology is particularly critical because it can process large amounts of data collected from various sensors and data sources. These technologies include statistical analysis, predictive models, machine learning, and data mining, which aim to optimize traffic flow management, improve safety, and enhance user experience by refining data.

[0003] In the existing technology, there is a lack of analysis on the frequency of data access, and effective data classification cannot be achieved, which results in the inability of the storage system to reasonably allocate resources according to the importance of data, thus affecting the optimization of data retrieval efficiency and storage costs. At the same time, there is also a lack of flexibility in data storage management, and the data location cannot be dynamically adjusted according to the actual access mode, which may cause access delays during peak hours, reduce user satisfaction, and increase potential risks and costs. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose an artificial intelligence big data processing method and system for smart transportation.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme, an artificial intelligence big data processing method for smart transportation, comprising the following steps:

[0006] Collect real-time traffic data and remove erroneous or incomplete records, classify and mark frequently accessed data, and generate cleaning and classification results; based on the cleaning and classification results, use machine learning to learn and analyze data access patterns to obtain a data access frequency model;

[0007] Based on the data access frequency model, high-frequency access data is allocated to a solid-state drive or a random access memory, and low-frequency access data is migrated to a mechanical hard disk to generate a hierarchical storage configuration; based on the hierarchical storage configuration, the real-time requirements of the data are analyzed, and the dynamic position of the data between each storage level is adjusted to obtain an optimized storage position;

[0008] According to the optimized storage location, cache data that has not been accessed for a long time is cleared to obtain a cache optimization result;

[0009] Migrate data with reduced access frequency from the solid-state drive or random access memory to the mechanical hard disk to generate data migration efficiency results; adjust the storage hierarchy and resource allocation strategy based on the data migration efficiency results to obtain storage structure optimization results.

[0010] Preferably, the steps of obtaining the cleaning and classification results are:

[0011] Collect real-time traffic data, check data completeness and accuracy, remove erroneous or incomplete records, and obtain preliminary cleansed data;

[0012] Based on the preliminary cleansing data, the traffic data is frequently marked, and frequently accessed and infrequently accessed data are distinguished and marked to obtain a cleaning and classification result.

[0013] Preferably, the steps of acquiring the data access frequency model are:

[0014] Based on the cleaning and classification results, a decision tree algorithm is used to configure the depth of the tree and the minimum number of samples for splitting nodes, and a model is trained to obtain a preliminary model;

[0015] According to the preliminary model, optimizing the model complexity by pruning to obtain an optimized model;

[0016] Based on the optimized model, the data access frequency model is verified to obtain the data access frequency model.

[0017] Preferably, the steps of obtaining the hierarchical storage configuration are:

[0018] Based on the data access frequency model, the data set is analyzed to identify the access frequency of each data point, and by performing frequency analysis on each type of data, high-frequency access data and low-frequency access data are distinguished to obtain data access frequency marks;

[0019] According to the data access frequency mark, the data storage priority is calculated, and the calculation formula is:

[0020]

[0021] Among them, P i represents the storage priority of data item i, F i represents the access frequency of data item i, w k represents the weight factor, c k represents the context feature factor associated with data access, and K is the total number of context features;

[0022] Based on the storage priority and combined with the data access frequency mark, high priority data is directed to the solid state drive or random access memory, and low priority data is migrated to the mechanical hard disk to obtain a hierarchical storage configuration.

[0023] Preferably, the step of obtaining the optimized storage location is:

[0024] Based on the hierarchical storage configuration, the real-time requirements of each data item are evaluated, the access records and time sensitivity of each data item are analyzed, each data item is classified into high, medium and low real-time sensitivity levels, and the real-time requirement analysis of each data item is obtained;

[0025] According to the real-time demand analysis, the priority adjustment index of the data item in the storage layer is calculated. The calculation formula is:

[0026]

[0027] Among them, ZP i is the priority adjustment index of data item i, H i Score the real-time requirement of data item i, L i is the loading frequency of data item i, M is the maximum loading frequency, a i ,b i ,c i is the adjustment factor;

[0028] Based on the priority adjustment index, the position of each data item in storage is adjusted, and high-priority data items are allocated to storage locations around computing hardware to obtain optimized storage locations.

[0029] Preferably, the steps of obtaining the cache optimization result are:

[0030] Based on the optimized storage location, analyzing the last access time of the data item, comparing it with the current time, calculating the time difference of each data item since the last access, and obtaining a non-access time list;

[0031] According to the unaccessed time list, the data clearing weight is calculated using the formula:

[0032]

[0033] Among them, W i represents the clearing weight of data item i, Δt i is the time since the last access of data item i, α is the adjustment sensitivity coefficient, and τ is the time threshold;

[0034] Based on the clearing weight, it is determined whether each data item is to be cleared, and data with a weight lower than a set threshold is cleared to obtain a cache optimization result.

[0035] Preferably, the steps for obtaining the data migration efficiency result are:

[0036] Analyze the data access log in the solid state drive or random access memory, identify the data items with reduced access frequency by comparing the past access frequency of each data item with the current access frequency, and obtain a list of data items with reduced frequency;

[0037] Based on the frequency reduction data list, the necessity score of data migration is calculated using the formula:

[0038]

[0039] Among them, N i represents the necessity score of data item i, r base is the baseline access frequency, r current is the current access frequency, V i is the volume of data item i;

[0040] Based on the necessity score, data items are selected to be migrated to the mechanical hard disk to obtain data migration efficiency results.

[0041] Preferably, the steps of obtaining the storage structure optimization result are:

[0042] Analyze the data migration efficiency results, identify the storage tiers with the lowest and highest efficiency, determine the data and resources that need to be adjusted, and obtain a preliminary adjustment plan;

[0043] Based on the preliminary adjustment plan, storage resources and levels are reconfigured, data storage locations and access paths are optimized, and storage structure optimization results are obtained.

[0044] The present invention provides an artificial intelligence big data processing system, comprising:

[0045] The data collection and preprocessing module receives real-time traffic data, removes erroneous or incomplete records, and marks frequently accessed data to obtain data preparation results;

[0046] The data analysis and model building module uses machine learning technology to learn and analyze data access patterns based on data preparation results and build a data access frequency model;

[0047] The data storage strategy configuration module uses the access frequency model to allocate high-frequency access data to solid-state drives or random access memories, and migrate low-frequency access data to mechanical hard drives to establish a hierarchical storage configuration;

[0048] The dynamic storage optimization module analyzes the real-time requirements of data based on the hierarchical storage configuration, adjusts the location of data between storage levels, and clears unaccessed cache data after optimizing the storage location;

[0049] The data migration and resource adjustment module migrates data from the solid-state drive or random access memory to the mechanical hard disk according to the data characteristics with reduced access frequency, and adjusts the storage level and resource allocation strategy.

[0050] Compared with the prior art, the advantages and positive effects of the present invention are:

[0051] In the present invention, real-time data collection and cleaning removes erroneous or incomplete records to ensure data accuracy. Then, the data is intelligently classified to distinguish between frequent and low-frequency access data, thereby improving the efficiency of data access. Then, machine learning is used to analyze these data and build an access frequency model to provide a scientific basis for optimizing data storage locations. In addition, by dynamically adjusting the location of data between different storage levels, the allocation of storage resources is optimized, the overall efficiency of the data processing system is improved, and operating costs are reduced, with high practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the steps of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] See also Figure 1 The present invention provides a technical solution, an artificial intelligence big data processing method for smart transportation, comprising the following steps:

[0055] Collect real-time traffic data and remove erroneous or incomplete records, classify and mark frequently accessed data, and generate cleaning and classification results; based on the cleaning and classification results, use machine learning to learn and analyze data access patterns to obtain a data access frequency model;

[0056] Based on the data access frequency model, high-frequency access data is allocated to solid-state drives or random access memories, and low-frequency access data is migrated to mechanical hard drives to generate hierarchical storage configurations. Based on the hierarchical storage configurations, the real-time requirements of data are analyzed, and the dynamic position of data between each storage level is adjusted to obtain the optimized storage location.

[0057] According to the optimized storage location, cache data that has not been accessed for a long time is cleared to obtain cache optimization results;

[0058] Migrate data with reduced access frequency from the solid-state drive or random access memory to the mechanical hard disk to generate data migration efficiency results; adjust the storage hierarchy and resource allocation strategy based on the data migration efficiency results to obtain storage structure optimization results.

[0059] The steps to obtain the cleaning and classification results are:

[0060] Collect real-time traffic data, check data completeness and accuracy, remove erroneous or incomplete records, and obtain preliminary cleansed data;

[0061] Based on the preliminary cleaned data, the traffic data is frequently marked, and the frequently and infrequently accessed data are distinguished and marked to obtain the cleaning and classification results.

[0062] Specifically, in order to collect basic data information in traffic scenes, video surveillance equipment and flow detection devices are set up and continuous recording is performed according to the set frequency. The timestamp format in the recorded data is defined and unified into the form of year-month-day hour: minute: second. The geographic coordinate range in the urban road management database is called as a reference and compared with the collected longitude and latitude values. If the longitude and latitude exceed the range, it is regarded as erroneous data and removed. The range is defined by the transportation department and registered before collection. The speed value and acceleration value also need to be compared with the preset range, for example, the speed is compared with the range of 0 to 60 meters per second and the acceleration is compared with the range of 0 to 1 The interval comparison is performed at 0 meters per second squared. These interval limits are set according to experience and local road conditions and a comparison table is established before collection. The integrity is checked by traversing each record in turn, including timestamp, geographic coordinates, speed, acceleration and other fields, and detecting whether there are empty or abnormal values. If a field is missing, it is considered an incomplete record. If it is detected that it does not match the valid range, it is determined to be an abnormal record and deleted. For boundary values, they can be confirmed one by one in combination with vehicle type and road speed limit standards to eliminate suspected invalid data. Finally, the records that meet the requirements of completeness and accuracy are numbered and stored as a group of data, namely, preliminary cleaned data.

[0063] After obtaining the preliminary cleaning data obtained above, first define a fixed period for counting the total number of statistical accesses. For example, with a 24-hour period, retrieve the data items with the same identifier in the data set one by one and count the number of times they are accessed within a preset time period. To set the threshold for distinguishing frequent and infrequent accesses, refer to the traffic signal optimization standard and the actual road traffic conditions and select a specific value based on experience. For example, if the number of accesses is greater than 50 times within 24 hours, it is considered a frequent access. In the process of counting the number of accesses, it is necessary to effectively aggregate the data items with the same identifier, check for duplicate records, and ensure that the access time falls within the statistical period. Records that exceed the threshold are uniformly marked as high-frequency access data and their identifiers and access times are recorded. Data items that do not reach the threshold are marked as low-frequency accesses and the identifiers and access times are also recorded. The above-mentioned labeling results are summarized to obtain the cleaning and classification results.

[0064] The steps to obtain the data access frequency model are:

[0065] Based on the cleaning and classification results, the decision tree algorithm is used to configure the tree depth and the minimum number of samples for splitting nodes, and the model is trained to obtain a preliminary model;

[0066] According to the preliminary model, the model complexity is optimized by pruning to obtain the optimized model;

[0067] Based on the optimized model, the data access frequency model is verified to obtain the data access frequency model.

[0068] Specifically, after receiving the cleaning and classification results obtained above, the data with high frequency and low frequency marked are used as the training set. The training set is first divided into several batches according to the urban road traffic range and multi-time access requirements and gradually input into the training program. Then, the depth of the decision tree is set to an empirical upper limit. For example, the maximum depth can be set to 10. This value comes from the estimation of the complexity of the traffic data and can be appropriately adjusted according to the changes in daily traffic flow. Then, the minimum number of samples for the split node is defined. For example, the minimum number of samples can be between 5 and 10 and set by the average value obtained from the previous traffic statistics. During the training process, the classification error is calculated for each split node and it is evaluated whether to continue splitting. If the amount of data in the node is lower than the preset minimum sample number threshold, the split will not continue. The threshold is consistent with the range selected above and refers to the scale data of multiple batches of samples. Through multiple rounds of iterative calculations, the decision tree gradually establishes each classification path, the node division method is cumulatively recorded and compared, and excessive splitting or obvious deviation is excluded. Finally, all split nodes are integrated and the tree structure of the current stage, i.e., the preliminary model, is output.

[0069] After obtaining the preliminary model obtained above, the relevant indicators for pruning are calculated based on the current tree structure of the model and a complexity penalty factor α is set. The specific value of this factor can be taken between 0.01 and 0.05 and dynamically selected according to the classification error rate and node size in the training set. Then, the total amount of the corresponding error plus the penalty term under different α is gradually compared. If several consecutive iterations converge, it means that the α is adapted to the current data set. When judging the importance of each node, the classification of the node in different time periods is checked and the actual error rate is observed to see whether it exceeds an empirical threshold. The empirical threshold is set by the confusion between high-frequency and low-frequency data and can be selected between 5% and 10%. If some nodes are found to contribute less to the overall prediction, they will be pruned. Pruning is performed by cyclically checking and recursively removing these nodes. The above operation is repeated until the structural complexity cannot be further reduced, and then the optimized model is obtained.

[0070] After obtaining the optimized model obtained above, the model is verified in combination with a sample set specifically used for verification. First, a part of the high-frequency data and low-frequency data are divided for verification respectively. An accuracy threshold based on actual traffic conditions is selected to determine whether the prediction error of the model is acceptable. The threshold can be set between 60% and 80% according to road capacity and peak period characteristics. When verifying, the classification accuracy of the model for high-frequency data and low-frequency data is statistically analyzed and the differences are compared. After multiple iterations, if the overall error is lower than the above accuracy threshold, it can be considered that the current model can adapt to the access distribution of traffic data. If there is a significant deviation in the high-frequency or low-frequency prediction results, the sub-node segmentation method will be adjusted and re-verified until the error meets the threshold range. After confirming that all indicators meet the requirements, the model is named data access frequency model.

[0071] The steps to obtain the hierarchical storage configuration are:

[0072] Based on the data access frequency model, the data set is analyzed to identify the access frequency of each data point. By performing frequency analysis on each type of data, high-frequency access data and low-frequency access data are distinguished to obtain data access frequency markers.

[0073] According to the data access frequency mark, the data storage priority is calculated, and the calculation formula is:

[0074]

[0075] Among them, P i represents the storage priority of data item i, F i represents the access frequency of data item i, w k represents the weight factor, c k represents the context feature factor associated with data access, and K is the total number of context features;

[0076] Based on storage priority and combined with data access frequency marking, high-priority data is directed to the solid-state drive or random access memory, and low-priority data is migrated to the mechanical hard disk to obtain a hierarchical storage configuration.

[0077] Specifically, based on the data access frequency model obtained above and reading the historical access records in the traffic database, the unique identifier of each record is associated with its corresponding access time, and a statistical time window is set to accumulate the number of access requests for different data points in the record. When comparing the fluctuation range of urban road traffic, the data points with the same identifier are aggregated and the cumulative number of requests for each data point in the time window is compared. If the cumulative number of requests for some data points is found to be higher than 10 times per day, it means that the access is relatively active. The range value is the median level selected after collecting access counts for 30 consecutive days in actual monitoring and combined with different time periods. The peak value is used to set the access times. When sorting the cumulative times, the identifiers that are more concentrated and far above the average level will be classified as high-frequency access categories, and the identifiers that are equal to or much lower than the median will be classified as low-frequency access categories. During this period, it is necessary to check each item one by one to identify whether the access time falls within the statistical time window and eliminate data points that are not in this window. Requests that cross dates or time periods are accumulated into the counts of the corresponding time periods. Through such classification, the number of accesses to each piece of data within the statistical time window can be obtained. Subsequently, these number information is merged and each identifier is labeled with the corresponding frequency category. Finally, the high-frequency access or low-frequency access labels of all identifiers are integrated to obtain the data access frequency mark.

[0078] The benefit of the formula is that it takes into account the coupling relationship between access frequency and contextual feature factors and balances the priority of data in different situations through the combination of exponentials and radicals. i The acquisition steps are as follows: first, count the data points with the same identifier in the traffic flow database, and add one whenever an access request to the identifier is detected. Then, the total number of accesses to the identifier is accumulated and summed up during the observation period of 15 consecutive days to obtain the total number of accesses to the identifier. For example, if the number of accesses to a data point within 15 days is 8, then F i =8,w k The acquisition step is to record the importance of each context feature as the corresponding weight after considering the traffic data in multiple dimensions in the early stage. For example, when monitoring urban traffic, if the vehicle type (small car, bus, truck, etc.) is considered to be more critical, its weight is larger. In actual operation, the frequency of occurrence of each feature can be quantified by aggregating the historical traffic data, so as to obtain a set of weight sequences w1 = 0.35 and w2 = 0.20, etc., c kThe acquisition method is to convert non-numerical context features into numerical results. For example, different lanes are numbered 1 to 5 to indicate the distinction of the lane area and are superimposed with the morning peak and evening peak time coefficients. Set c1=3 to represent the section of the morning peak lane number 3, c2=2 to represent the section of the non-peak lane number 2, and K=2 to select two context feature factors;

[0079] Calculation process:

[0080] Compute the summation in the denominator:

[0081]

[0082] Substituting into the denominator:

[0083]

[0084] Calculate the numerator:

[0085]

[0086] The final calculation P i :

[0087]

[0088] The result shows that for an access count of 8 and after defining the above weights and contextual feature factors, the final storage priority of this data point is about 1903.80. If another data point is subsequently encountered and the calculated value is larger, it means that its priority is higher. If the value is smaller, it means that the priority is relatively low.

[0089] After obtaining the storage priority and data access frequency tags obtained above, it is necessary to allocate all high-priority data records. First, select the solid-state drive or random access memory as the main storage medium and roughly divide the overall storage capacity. For example, in the traffic flow monitoring system, 40% of the total capacity can be allocated to store records that are accessed multiple times in a short period of time. This ratio is based on the average value range confirmed after the access frequency statistics obtained above. Then, sort them according to the capacity size and access density of each high-priority record, and install the high-frequency data with an access density higher than 15 times a day first. The data is loaded and occupies the priority storage area. The data accessed between 10 and 15 times a day is also summarized into the solid-state drive or random access memory. As for the low-priority data, if the statistical analysis shows that the number of daily accesses is less than 10 times, it is judged as low frequency. It is packaged and transferred to the mechanical hard disk according to the original space order. When implementing the transfer, it is checked section by section according to the data volume and transmission speed and the transmission results are recorded. After confirming again that there is no missed transmission or wrong transmission, the final merge configuration is completed. Finally, the location of the allocated high-priority data and the corresponding low-priority data movement information are uniformly registered to obtain the hierarchical storage configuration.

[0090] The steps to obtain the optimized storage location are:

[0091] Based on the hierarchical storage configuration, evaluate the real-time requirements of each data item, analyze the access records and time sensitivity of each data item, classify each data item into high, medium and low real-time sensitivity levels, and obtain the real-time requirements analysis of each data item;

[0092] According to the real-time demand analysis, the priority adjustment index of the data item in the storage layer is calculated. The calculation formula is:

[0093]

[0094] Among them, ZP i is the priority adjustment index of data item i, H i Score the real-time requirement of data item i, L i is the loading frequency of data item i, M is the maximum loading frequency, a i , b i , c i is the adjustment factor;

[0095] Based on the priority adjustment index, the position of each data item in the storage is adjusted, and high-priority data items are allocated to storage locations around the computing hardware to obtain optimized storage locations.

[0096] Specifically, after obtaining the hierarchical storage configuration obtained above, the access time of different data items in road monitoring is recorded and the corresponding access times are read, and the time sensitivity covered by these records is analyzed one by one. If some data items contain multiple access requests in adjacent time periods during the collection process, they are considered to have high timeliness requirements, and need to be further combined with the data of the vehicle flow sensing device to confirm factors such as vehicle type and driving speed. For example, the time period is divided into the morning peak period of 07:00 to 09:00 and the evening peak period of 17:00 to 19:00, and the number of accesses in each peak period is cumulatively compared. If a data item is continuously accessed during the above peak periods, it can be classified as highly real-time sensitive. Level. If there are a small number of accesses during non-peak hours such as 22:00 at night to 05:00 the next day, it can be marked as a medium real-time sensitivity level. If there are very few accesses only in occasional periods, it will be marked as a low real-time sensitivity level. These classification boundary values ​​are obtained by referring to the historical monitoring data of urban traffic. For example, after 90 days of continuous monitoring, the average and median of the data access volume in multiple peak periods are counted, and the situation where it is higher than the average or deviates significantly from the median is set as high real-time, between the two is medium real-time, and below the median by a considerable proportion is classified as low real-time. Then the classification label of each data item is recorded. Finally, after confirming the three real-time sensitivity levels of high, medium and low, the real-time demand analysis of each data item is formed.

[0097] The formula is useful in that it takes into account the real-time demand score of data items and the actual loading frequency in the form of a power exponent, and uses the maximum loading frequency for normalization. This can reflect the priority of different data items within the storage hierarchy in a variety of scenarios.

[0098] H i The parameter acquisition step is to combine the time delay range and access urgency in vehicle traffic monitoring, first count the response delay caused by the data item being accessed at different times of the day and night, divide the delay value into multiple intervals from 0 seconds to 5 seconds, 5 seconds to 15 seconds, 15 seconds to 30 seconds, etc., extract the corresponding delay times, and then weight these times to obtain a real-time demand score such as H i =7;

[0099] L i The parameter acquisition step is to count the number of times the data item is loaded in the last 30 days, and record the triggering event of the loading operation at each access. For example, the data item is read after a vehicle is detected passing through the monitoring area. The total number of daily triggers is summed up and then divided by 30 to obtain the average loading frequency. If the statistical result is 45 times, it is recorded as L i =45;

[0100] The step of obtaining the M parameter is to select the maximum loading frequency of all data items within the above statistical range. For example, if the highest loading frequency found in the same monitoring system is 60 times, it is recorded as M = 60;

[0101] a i 、b i 、c i is the adjustment coefficient for different types of data. Its value is obtained by analyzing indicators such as data latency tolerance, repeated reading tendency, and resource burden. For example, for extremely sensitive data items, a i If the actual evaluation confirms that this item is too high, you can set a i =1.2, for scenarios where the loading frequency has a significant impact, b i =1.1, for the case where the maximum loading frequency affects the relative balance, c i =0.8;

[0102] Calculation process:

[0103] Process the molecule and calculate H i ai :

[0104] 7 1.2 ≈7 1.2 ≈15.17

[0105] Simultaneous calculation

[0106] 45 1.1 ≈45 1.1 ≈82.84

[0107] Multiply the two together to get the numerator:

[0108] 15.17×82.84≈1257.47

[0109] Calculate the denominator M ci :

[0110] 60 0.8 ≈60 0.8 ≈20.45

[0111] Final calculation ZP i :

[0112]

[0113] The results show that for a data item with a real-time requirement score of 7 and an average loading frequency of 45 times, the priority adjustment index obtained after the above adjustment coefficient calculation when the maximum loading frequency is 60 times is approximately 61.48. If another data item is subsequently detected to have a calculation result higher than 61.48, it means that it has a higher priority in terms of real-time requirements and loading frequency. If the result is lower than 61.48, it means that its priority is relatively low.

[0114] After calculating the priority adjustment index obtained above, it is necessary to compare each data item one by one in combination with the existing storage space layout in the actual system. If it is found that some data items have a high access density in the peak period statistics and the real-time demand score ranks high in the data obtained above, they can be determined as high-priority data items. Subsequently, the available space of the solid-state drive or random access memory will be checked first, and the positions of these data items will be allocated one by one from the perspective of capacity allocation. For example, 30% of the space of the storage medium will be reserved and the data items with a priority index of more than 60 obtained above will be concentrated in the peripheral positions to reduce response delay. For data items with an index between 40 and 60, they can be placed in sequence according to similar features. If some data items are only accessed in small quantities late at night and the index is lower than 40, they will be included in the general storage area. When the overall allocation is completed, the index table will be checked again in the established order to track the allocation path of each data item. If it is detected that it does not match the access sensitivity level defined above, the index table will be revised and overwritten on the corresponding medium. Finally, all the allocated data will be numbered as a whole to obtain the optimized storage location.

[0115] The steps to obtain cache optimization results are:

[0116] Based on the optimized storage location, analyze the last access time of the data item, compare it with the current time, calculate the time difference since the last access of each data item, and obtain the unaccessed time list;

[0117] According to the unaccessed time list, the data clearing weight is calculated using the formula:

[0118]

[0119] Among them, W i represents the clearing weight of data item i, Δt i is the time since the last access of data item i, α is the adjustment sensitivity coefficient, and τ is the time threshold;

[0120] Based on the clearing weight, decide whether to clear each data item, clear the data whose weight is lower than the set threshold, and get the cache optimization result.

[0121] Specifically, after obtaining the "optimized storage location" obtained above, it is necessary to read the last access time of each data item one by one. During the reading process, the time value is compared with the current system clock and the time difference is obtained. If the time length of a data item since the last access exceeds two days, the data item and the corresponding time difference are recorded. The value of two days is determined by access monitoring for one month. During this period, the access intervals of all data items are counted and the range with relatively concentrated distribution is found. Then, combined with the update cycle of the traffic data itself and the hardware storage capacity status, two days are determined as a key reference interval. After counting all time differences exceeding this interval, these time differences are sorted in sequence. For abnormal values, such as some data items with too large time differences or negative values, they are checked one by one to see if there are any record errors. The checking method can be combined with the index information obtained above and confirm the validity of the timestamp in the access log. For missing access records, they will be checked again and eliminated when comparing with the actual device operation log. After confirming the last access time of each data item and successfully obtaining the difference between it and the current time, these time differences are sorted into a list of unaccessed times, and the corresponding unique identifiers are assigned for subsequent association.

[0122] The benefit of the formula is that it introduces an exponential function to characterize the relationship between the non-access time of a data item and the removal weight, and makes the adjustment sensitivity coefficient and the time threshold jointly affect the removal decision;

[0123] Δt i The parameter acquisition step is to retrieve the time difference recorded in the "unaccessed time list" of the corresponding data item obtained previously. For example, if a certain data item is read and it has been 48 hours since the last access, let Δt i =48, this value is obtained by continuously monitoring and recording the access timestamp;

[0124] The step of obtaining the α parameter is to analyze the flexibility of data cleaning in the long-term test. By observing the number of data items cleared and the system memory utilization under different time differences, a regression analysis is performed on the collected time differences and the number of clearing operations after stabilization, and a better solution of about α = 0.10 is obtained;

[0125] The step of obtaining the τ parameter is to select a time threshold, which is determined to be 36 hours based on multiple storage management and operation and maintenance experiences. This experience comes from statistics of multiple rounds of operation and maintenance in the past. For example, after monitoring the average revisit cycle within 60 days and combining the device log to obtain the most intensive access interval distribution interval, the value near the distribution inflection point is set to 36 hours.

[0126] Calculation process:

[0127] Calculate the difference Δt in brackets i -τ:

[0128] 48-36=12

[0129] Multiply the result by -α:

[0130] -0.10×12=-1.2

[0131] Substitute the exponential form e -1.2 :

[0132] e -1.2 ≈0.301

[0133] Add this exponential result to 1 to get the denominator:

[0134] 1+0.301=1.301

[0135] Find the ratio of the numerator to the denominator:

[0136]

[0137] The results show that for data items that have been accessed for 48 hours, the clearing weight calculated by adjusting the sensitivity coefficient to 0.10 and the time threshold to 36 hours is approximately 0.768. When it is greater than 0.5, it means that the clearing weight of the data item is relatively high. If the clearing threshold defined subsequently is around 0.7, the data item will be considered to have a greater probability of being retained. If the parameters are subsequently changed or the time difference continues to increase, the clearing weight will also change accordingly and affect subsequent judgments.

[0138] After obtaining the previously obtained purge weight, the entire data set needs to be compared sequentially. Each time a data item is scanned, its calculated purge weight value will be extracted and compared with the set threshold. Data items below the threshold are considered to be targets that can be directly deleted. In the process of setting the threshold, a suitable level will be selected from the historical memory space utilization collected previously. For example, the memory usage is checked regularly every day within a month and the interval average of multiple capacity warnings is sorted out. If the system runs more tightly when the capacity is close to 80%, the threshold can be appropriately raised to about 0.75. Then all data items are traversed and the current purge weight and access identifier are recorded in sequence. If a value is lower than 0.75, it will be marked and deleted synchronously, and the corresponding storage location will be released to ensure subsequent merging. At the same time, the deleted data will be invalidated in the overall index table and the deletion record will be added in the access log. If data with a weight higher than 0.75 is encountered, it will continue to be retained. Finally, after all data items have been scanned, the deletion situation will be combined with the remaining storage space list to obtain the cache optimization result.

[0139] The steps to obtain the data migration efficiency results are as follows:

[0140] Analyze the data access log in the solid state drive or random access memory, identify the data items with reduced access frequency by comparing the past access frequency of each data item with the current access frequency, and obtain a list of data items with reduced frequency;

[0141] Based on the frequency reduction data list, the necessity score of data migration is calculated using the formula:

[0142]

[0143] Among them, N i represents the necessity score of data item i, r base is the baseline access frequency, r current is the current access frequency, V i is the volume of data item i;

[0144] Based on the necessity score, data items are selected to be migrated to the mechanical hard disk to obtain the data migration efficiency results.

[0145] Specifically, before analyzing the data access logs in the solid-state drive or random access memory, it is necessary to first centrally read the historical records of past access frequencies and current access frequencies, match and summarize each access record with the corresponding timestamp, and ensure that all data items can be correctly associated with their own access times. When classifying, you can first set a statistical period, such as selecting the access data of the last 30 days plus the new data of the current 7 days, and compare the access frequencies of these data one by one. If it is found that the number of accesses in the new period is less than the number of accesses in the old period and the difference is obvious, it is considered that the access frequency has decreased, and it needs to be reconfirmed in combination with vehicle traffic characteristics or video detection trigger records. If there are duplicate or blank marks in the record, they will be removed one by one and logged. The corresponding abnormal situations are recorded. During the comparison process, the access frequency that is 30% or more lower than the average value of the previous cycle can be used as the basis for judgment. This 30% threshold is determined in combination with the previously obtained access volume distribution. For example, when counting data within 60 days, the number of accesses to all data items per day is summed up and the average is calculated. Then, when comparing the ratio of the latest cycle to the old cycle of each data item, if the newer cycle decreases by more than 30% compared with the old cycle, it can be classified as a significant decrease in frequency. Then, according to its importance in the specific business scenario, the qualified data items are recorded in the "frequency decrease data list". Finally, all such data items are summarized and compared with the original access log to obtain the sorted frequency decrease data list.

[0146] The benefit of the formula is that it uses the logarithmic function and the cube root of the data volume to jointly incorporate the degree of decrease in access frequency and the data volume into the evaluation, thereby providing a graded measurement for data items of different sizes;

[0147] r baseThe step of obtaining the parameter is to read the reference access count of each data item in the previously determined "frequency reduction data list" in the old cycle. For example, after observing the logs of the last 30 days, the average access frequency of a data item in the old cycle is counted as 100 times, then r base =100;

[0148] r current The parameter acquisition step is to combine the number of visits in the latest cycle of the same data item, and analyze the cumulative number of visits in the last 7 days or other monitoring periods. For example, if the actual statistics is 70 times, then r current =70;

[0149] V i The step of obtaining the parameter is to read the volume of the data item and convert it into a unified unit. For example, in the monitoring system, the size of the image or video frame data is converted from bytes to megabytes. If the record shows that the volume of this data item is 8MB, then V i =8;

[0150] Calculation process:

[0151] Calculate the sum of the frequency differences within the brackets:

[0152] 1+r base -r current =1+100-70=31

[0153] Take the logarithm of this result:

[0154] ln(31)≈3.434

[0155] Compute the cubic root of the volume:

[0156]

[0157] Divide the logarithm by the cubic root:

[0158]

[0159] The results show that for data items with an access frequency of 100 times in the old cycle, an access frequency of 70 times in the new cycle, and a data volume of 8MB, the necessity score is approximately 1.717. If data items are found in the same batch later with calculation results higher than 1.717, it means that they are more urgently in need of migration evaluation. If the results are lower than 1.717, it means that their degree of necessity is relatively reduced.

[0160] After obtaining the necessity score obtained above, it is necessary to make a migration judgment for each data item. First, list the scores of all data items in the frequency reduction data list obtained above, and then sort them from high to low. If the score is greater than a certain threshold, it will be regarded as a higher migration priority. The value of the threshold can be comprehensively confirmed based on the statistical results of previous storage management. For example, during the 90-day operation and maintenance process, observe the changes in storage space utilization and access frequency distribution and extract a score distribution curve. When the score is approximately in the range of 1.5 to 2.0, the corresponding migration effect is more reasonable. Therefore, 1.5 is set as the threshold for judging migration priority, and the migration strategy is executed for data greater than 1.5 in turn. In specific implementation, it is necessary to mark these data items, check their positions in the solid-state drive or random access memory, and then call segmentation or block method for physical movement. Finally, record the migration process and time again, summarize the completion status of the movement of this batch of data items, and obtain the final data migration efficiency result.

[0161] The steps to obtain the storage structure optimization results are:

[0162] Analyze the data migration efficiency results, identify the storage tiers with the lowest and highest efficiency, determine the data and resources that need to be adjusted, and obtain a preliminary adjustment plan;

[0163] Based on the preliminary adjustment plan, reconfigure storage resources and levels, optimize data storage locations and access paths, and obtain storage structure optimization results.

[0164] Specifically, after obtaining the data migration efficiency results obtained previously, first select the relevant indicators of the three commonly used levels of solid-state drives, random access memories, and mechanical hard drives for horizontal comparison and record information such as transmission time, read latency, and access success rate. Match this information with the migration batch data obtained previously item by item. If it is detected that the transmission time is 40% or more higher than the usual average in a certain migration, it is marked as low efficiency. This 40% standard comes from the sorting and comparison of multiple rounds of historical migration records. When comparing, the length of time spent on each migration and the corresponding amount of data are associated. If too many resources are occupied or the speed is unstable, it will be marked to the least efficient storage level according to the allocation rules. Similarly, if the access success rate is observed to be much higher than the average level in certain periods and the transmission time is kept in a reasonable range, such as not exceeding 10% of the usual average, then this level will be recorded as the most efficient level, and the corresponding data will also be considered when making the judgment. The access frequency and latency requirements of the data items are combined for re-screening. For example, if a level always maintains an average access latency of less than 1 second during the monitoring period, it means that its speed is relatively fast. After comprehensive processing of multiple batches of samples, these evaluation values ​​are sorted and the performance ranges of all levels are noted. According to the evaluation results, the lowest and highest efficiency level ranges are finally identified. Then, the data volume and access logs involved in these two levels are screened to confirm whether additional supporting resources are needed to make the two levels more balanced in future reading and writing. If it is found that some batch data occupies storage locations both in the low-efficiency level and in the high-efficiency level, the corresponding access cycle and peak period are recorded. Combined with the transmission rate and occupied space statistics obtained previously, the data that needs to be focused on is sorted out, and whether the hardware resources can support the subsequent access peak is also calculated. Finally, the above comparison results are summarized into a list of "data and resources that need to be adjusted" to form a preliminary adjustment plan.

[0165] After obtaining the preliminary adjustment plan obtained above, it is necessary to make a statistical comparison of the capacity and read-write characteristics of the solid-state drive, random access memory and mechanical hard disk from a macro perspective. By traversing the block size occupied by different data in the existing storage media and its access peak time, it is determined which medium has a higher load in what time period based on the accumulated access density over multiple days. For example, you can first check whether there is an obvious surge in concurrent access to the random access memory during the morning traffic rush hour of 07:00 to 09:00. If the access volume continues to approach the read-write threshold of this medium, it means that the medium needs to release or reallocate space. For other content with an application frequency of less than 10 times a day, it is guided to Mechanical hard disks are used to reduce high concurrency pressure. If the cumulative utilization rate on the solid-state hard disk exceeds a preset threshold, such as 80%, specific data also needs to be migrated to other media with higher idleness. After completing the overall configuration of the above resources and space, each high-priority data will be checked by path, and the access delay will be compared one by one to see if it has dropped to an acceptable range. For abnormal points, they will be split to see whether the access peak is too occupied or there is a large amount of old data accumulated. Combined with the data volume information obtained previously, local relocation or exchange storage is performed. Finally, after confirming that there is no obvious congestion in the read and write channels of each key application, all media and their configuration information will be integrated to obtain the storage structure optimization result.

[0166] The present invention provides an artificial intelligence big data processing system, comprising:

[0167] The data collection and preprocessing module receives real-time traffic data, removes erroneous or incomplete records, and marks frequently accessed data to obtain data preparation results;

[0168] The data analysis and model building module uses machine learning technology to learn and analyze data access patterns based on data preparation results and build a data access frequency model;

[0169] The data storage strategy configuration module uses the access frequency model to allocate high-frequency access data to solid-state drives or random access memories, and migrate low-frequency access data to mechanical hard drives to establish a hierarchical storage configuration;

[0170] The dynamic storage optimization module analyzes the real-time requirements of data based on the hierarchical storage configuration, adjusts the location of data between storage levels, and clears unaccessed cache data after optimizing the storage location;

[0171] The data migration and resource adjustment module migrates data from the solid-state drive or random access memory to the mechanical hard disk according to the data characteristics with reduced access frequency, and adjusts the storage level and resource allocation strategy.

[0172] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. An artificial intelligence big data processing method for smart transportation, characterized in that: The following steps are involved: Collect real-time traffic data and remove erroneous or incomplete records, classify and mark frequently accessed data, and generate cleaning and classification results; Based on the cleaning and classification results, machine learning is used to learn and analyze the data access pattern to obtain a data access frequency model; Based on the data access frequency model, high-frequency access data is allocated to a solid-state drive or a random access memory, and low-frequency access data is migrated to a mechanical hard disk to generate a hierarchical storage configuration; Based on the hierarchical storage configuration, the real-time requirements of the data are analyzed, and the dynamic position of the data between each storage level is adjusted to obtain an optimized storage position; According to the optimized storage location, cache data that has not been accessed for a long time is cleared to obtain a cache optimization result; Migrate data with decreasing access frequency from SSD or RAM to HDD to generate data migration efficiency results; The storage level and resource allocation strategy are adjusted based on the data migration efficiency result to obtain a storage structure optimization result.

2. The artificial intelligence big data processing method for smart transportation according to claim 1 is characterized in that: The steps for obtaining the cleaning and classification results are as follows: Collect real-time traffic data, check data completeness and accuracy, remove erroneous or incomplete records, and obtain preliminary cleansed data; Based on the preliminary cleansing data, the traffic data is frequently marked, and frequently accessed and infrequently accessed data are distinguished and marked to obtain a cleaning and classification result.

3. The artificial intelligence big data processing method for smart transportation according to claim 1 is characterized in that: The steps for obtaining the data access frequency model are: Based on the cleaning and classification results, a decision tree algorithm is used to configure the depth of the tree and the minimum number of samples for splitting nodes, and a model is trained to obtain a preliminary model; According to the preliminary model, optimizing the model complexity by pruning to obtain an optimized model; Based on the optimized model, the data access frequency model is verified to obtain the data access frequency model.

4. The artificial intelligence big data processing method for smart transportation according to claim 1 is characterized in that: The steps for obtaining the hierarchical storage configuration are: Based on the data access frequency model, the data set is analyzed to identify the access frequency of each data point, and by performing frequency analysis on each type of data, high-frequency access data and low-frequency access data are distinguished to obtain data access frequency marks; According to the data access frequency mark, the data storage priority is calculated, and the calculation formula is: Among them, P i represents the storage priority of data item i, F i represents the access frequency of data item i, w k represents the weight factor, c k represents the context feature factor associated with data access, and K is the total number of context features; Based on the storage priority and combined with the data access frequency mark, high priority data is directed to the solid state drive or random access memory, and low priority data is migrated to the mechanical hard disk to obtain a hierarchical storage configuration.

5. The artificial intelligence big data processing method for smart transportation according to claim 1 is characterized in that: The steps for obtaining the optimized storage location are: Based on the hierarchical storage configuration, the real-time requirements of each data item are evaluated, the access records and time sensitivity of each data item are analyzed, each data item is classified into high, medium and low real-time sensitivity levels, and the real-time requirement analysis of each data item is obtained; According to the real-time demand analysis, the priority adjustment index of the data item in the storage layer is calculated. The calculation formula is: Among them, ZP i is the priority adjustment index of data item i, H i Score the real-time requirement of data item i, L i is the loading frequency of data item i, M is the maximum loading frequency, a i ,b i ,c i is the adjustment factor; Based on the priority adjustment index, the position of each data item in storage is adjusted, and high-priority data items are allocated to storage locations around computing hardware to obtain optimized storage locations.

6. The artificial intelligence big data processing method for smart transportation according to claim 1 is characterized in that: The steps for obtaining the cache optimization result are: Based on the optimized storage location, analyzing the last access time of the data item, comparing it with the current time, calculating the time difference of each data item since the last access, and obtaining a non-access time list; According to the unaccessed time list, the data clearing weight is calculated using the formula: Among them, W i represents the removal weight of data item i, Δt i is the time since the last access of data item i, α is the adjustment sensitivity coefficient, and τ is the time threshold; Based on the clearing weight, it is determined whether each data item is to be cleared, and data with a weight lower than a set threshold is cleared to obtain a cache optimization result.

7. The artificial intelligence big data processing method for smart transportation according to claim 1 is characterized in that: The steps for obtaining the data migration efficiency result are as follows: Analyze the data access log in the solid state drive or random access memory, identify the data items with reduced access frequency by comparing the past access frequency of each data item with the current access frequency, and obtain a list of data items with reduced frequency; Based on the frequency reduction data list, the necessity score of data migration is calculated using the formula: Among them, N i represents the necessity score of data item i, r base is the baseline access frequency, r current is the current access frequency, V i is the volume of data item i; Based on the necessity score, data items are selected to be migrated to the mechanical hard disk to obtain data migration efficiency results.

8. The artificial intelligence big data processing method for smart transportation according to claim 1 is characterized in that: The steps for obtaining the storage structure optimization result are: Analyze the data migration efficiency results, identify the storage tiers with the lowest and highest efficiency, determine the data and resources that need to be adjusted, and obtain a preliminary adjustment plan; Based on the preliminary adjustment plan, storage resources and levels are reconfigured, data storage locations and access paths are optimized, and storage structure optimization results are obtained.

9. An artificial intelligence big data processing system according to the artificial intelligence big data processing method for smart transportation according to any one of claims 1 to 8, characterized in that: include: The data collection and preprocessing module receives real-time traffic data, removes erroneous or incomplete records, and marks frequently accessed data to obtain data preparation results; The data analysis and model building module uses machine learning technology to learn and analyze data access patterns based on data preparation results and build a data access frequency model; The data storage strategy configuration module uses the access frequency model to allocate high-frequency access data to solid-state drives or random access memories, and migrate low-frequency access data to mechanical hard drives to establish a hierarchical storage configuration; The dynamic storage optimization module analyzes the real-time requirements of data based on the hierarchical storage configuration, adjusts the location of data between storage levels, and clears unaccessed cache data after optimizing the storage location; The data migration and resource adjustment module migrates data from the solid-state drive or random access memory to the mechanical hard disk according to the data characteristics with reduced access frequency, and adjusts the storage level and resource allocation strategy.

Images