Intelligent traffic large-scale data optimization storage method

Abstract

The invention relates to the technical field of data storage, in particular to an intelligent traffic large-scale data optimization storage method. A smart traffic large-scale data optimization storage method comprises the following steps: S1, collecting historical data of a target person, and preprocessing the historical data; s2, respectively obtaining a holiday time sequence and a driving school reservation sequence of the target school by using an FIFO algorithm and an LFU algorithm; s3, allocating the reservation data according to the family position data and the examination place data of the target person; and S4, inputting the latest reservation data and the historical examination information of the target person into an examination time point prediction model to obtain an examination time point predicted by the target person. According to the invention, the reservation data is classified according to the similarity of the reservation of the target personnel at each examination site, the classified data is stored in a distributed manner, and the storage priority of the student data is detected and adjusted, so that reasonable allocation of resources is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a method for optimizing storage of large-scale data in smart transportation. Background Art

[0002] Traffic data includes not only road vehicle driving conditions, traffic flow, and accident records, but also indirect data such as driver's license deduction and bonus points, and driver's license test appointments.

[0003] With the development of intelligent transportation, the storage and management of test appointments and test data has become a complex and critical field. Especially in a large-scale data environment, the traditional static storage method cannot meet the needs of real-time data updates and dynamic storage, resulting in low efficiency in processing high-frequency appointment changes and test data retrieval, and it is difficult to fully guarantee the real-time and reasonable storage of data. At present, the commonly used distributed storage is used to store the information of examinees, and the examinees' information is stored in different servers. However, since the similarity in the examinees' information is not fully considered, all data is simply distributed and stored indiscriminately, which does not effectively improve the storage efficiency; and because the system cannot dynamically distinguish the importance of data, it may allocate unnecessary storage resources for low-priority data, especially after students return to school, it still occupies unnecessary storage resources. This phenomenon is more obvious, affecting the overall storage efficiency of the system. Summary of the invention

[0004] In order to overcome the shortcomings of indiscriminate distributed storage and low data storage efficiency when storing data on a server, the present invention provides a method for optimizing storage of large-scale data in smart transportation.

[0005] The technical implementation scheme of the present invention is: a method for optimizing and storing large-scale data of intelligent transportation, comprising the following steps: S1: Collect historical data of the target person and pre-process the historical data; S2: Use the FIFO algorithm and the LFU algorithm to obtain the target school's holiday time series and the driving school appointment sequence respectively; S3: Allocate the appointment data according to the target person’s home location data and test location data; S4: inputting the latest reservation data and historical examination information of the target person into the examination time point prediction model to obtain the predicted examination time point of the target person; S5: adjusting the priority of data storage according to the predicted test time point, the data storage priority formula and the storage priority adjustment formula; S6: Adjust the storage location of the target personnel data according to the adjusted data storage priority.

[0006] Preferably, the historical data of the target person is collected and preprocessed, including: obtaining the historical data of the target person, school holiday data and driving school appointment data, and performing data cleaning to fill in missing values, remove duplicate values, and remove outliers.

[0007] Preferably, the use of the FIFO algorithm and the LFU algorithm to obtain the target school's holiday time series and the driving school reservation sequence respectively includes: using the FIFO algorithm to input the school's holiday data that is greater than a preset number of days each year to obtain the target school's holiday time series, wherein the holiday time series includes a holiday start time and an end time; The driving school reservation data at preset time intervals are inputted using the LFU algorithm to obtain a driving school reservation sequence, wherein the preset number of times and the update time of each element are set when the LFU algorithm is used.

[0008] Preferably, the LFU algorithm is used to input the driving school reservation data at preset time intervals to obtain a driving school reservation sequence, wherein a preset number of times is set when the LFU algorithm is used, including: recording the time of each element in the driving school reservation sequence, and judging the driving school reservation sequence, When the number of new elements replaced reaches a preset number, a judgment is made based on the update time of the elements in the driving school reservation sequence. If there is an update time difference in the driving school reservation sequence that is greater than or equal to a preset time threshold, the replaced new element replaces the element with the earliest update time.

[0009] Preferably, the allocating the reservation data according to the home location data and the test location data of the target person includes: inputting the home location of the target person, the pass rate of the test location, and the locations that have been tested into a matching degree formula to obtain the matching degree between the target person and each test location, and allocating the reservation data of the target person to the test location with the maximum matching degree; The matching degree formula is: , In the formula, is the matching degree, is the first weight corresponding to the distance ratio, is the actual distance, is the maximum reference distance set, is the second weight corresponding to the pass rate, is the pass rate of the test location, is the third weight corresponding to the matching rate, is the number of matches of the completed test location in the driving school reservation sequence, Make reservations for all test locations in the sequence for the driving school.

[0010] Preferably, the latest appointment data and historical examination information of the target person are input into the examination time point prediction model to obtain the predicted examination time point of the target person, including: collecting historical appointment data, converting the input data into a time series format, and preprocessing the data in a standardized manner, using mean square error as a loss function, and finally obtaining a trained examination time point prediction model, wherein the examination time point prediction model is trained using a Transformer model, the latest appointment data and historical examination information of the target person are input into the examination time point prediction model, the predicted examination time point of the target person is obtained, and the predicted examination time point of the target person is judged.

[0011] Preferably, the latest appointment data and historical examination information of the target person are input into the examination time point prediction model to obtain the predicted examination time point of the target person, and the predicted examination time point of the target person is judged, including: when the predicted examination time point of the target person is not within the latest holiday start time and end time of the same label in the holiday time series, or the historical appointment data and historical examination information are less than a preset quantity threshold, the middle value of the latest holiday start time and end time in the target person's holiday time series with the same label is used as the predicted examination time point.

[0012] Preferably, the step of adjusting the data storage priority according to the predicted test time point, the data storage priority formula and the storage priority adjustment formula comprises: adjusting the historical reservation data using the data storage priority formula, wherein the data storage priority formula is: , in The priority of data storage is is the current time, For the predicted test time point, A parameter for adjusting the rate at which priority changes.

[0013] Preferably, the step of adjusting the data storage priority according to the predicted test time point, the data storage priority formula and the storage priority adjustment formula comprises: adjusting the data storage priority using the storage priority adjustment formula to obtain the adjusted data storage priority, wherein the storage priority adjustment formula is: .

[0014] in is the adjusted data storage priority. The priority of data storage is : is a function used to adjust the weight of storage priority according to the exam progress, is the adjustment coefficient, Indicates the time that has passed since registration.

[0015] Preferably, the storage location of the target personnel data is adjusted through the adjusted data storage priority, including: extracting the historical appointment data of the target personnel whose data storage priority is greater than or equal to the preset priority threshold into hot storage, and extracting the historical appointment data of the target personnel whose data storage priority is less than the preset priority threshold into cold storage.

[0016] Beneficial effects: The present invention classifies appointment data by utilizing the similarity of appointment data, and performs distributed storage according to the appointment data; uses FIFO and LFU algorithms to achieve effective management and storage of different types of data, so that commonly used data is stored first and infrequently used data is reasonably eliminated, thereby improving the utilization rate of storage space and data access speed; uses a prediction model to enable the system to more accurately predict the test time point, and then reasonably arrange appointments, avoid resource waste, and improve user experience; when the predicted test time point does not meet the expected conditions, the system can automatically adjust the prediction result, thereby enhancing the flexibility of the system and its adaptability to changes in the external environment; uses data storage priority formulas and adjustment formulas to dynamically adjust the storage priority of data according to the students' vacation situation, ensuring that important data can be quickly accessed and non-important data is stored in low-cost storage media. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a method for optimizing storage of large-scale intelligent transportation data according to the present invention; Figure 2 It is a schematic diagram of the holiday time series of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] A method for optimizing storage of large-scale data in smart transportation, such as Figure 1-Figure 2 As shown, the following steps are included: S1: Collect historical data of the target person and pre-process the historical data; Obtain the historical data, school holiday data, and driving school reservation data of the target personnel, and perform data cleaning to fill in missing values, remove duplicate values, and remove outliers.

[0020] It should be noted that the collected data includes the student's past appointment records (such as the time and location of each appointment), the school's holiday arrangements (such as the specific time of winter and summer vacations), and the test venue data for each appointment at the driving school. If a specific appointment record is missing, it is filled in by finding appointment records with similar dates, or using the mean interpolation method; if the same appointment is found to be recorded repeatedly in the data set, such as two appointments on the same day, the system will only retain one record; if the time of a certain appointment is very different from the usual appointment time (for example, a late-night appointment), the system will identify and remove these abnormal data by setting thresholds.

[0021] S2: Use the FIFO algorithm and the LFU algorithm to obtain the target school's holiday time series and the driving school appointment sequence respectively; Use the FIFO algorithm to input the school's holiday data that is greater than a preset number of days each year to obtain a holiday time series for the target school, wherein the holiday time series includes the holiday start time and end time; The driving school reservation data at preset time intervals are inputted using the LFU algorithm to obtain a driving school reservation sequence, wherein the preset number of times and the update time of each element are set when the LFU algorithm is used.

[0022] It should be noted that the FIFO algorithm is used: the holiday data of each year is input into the FIFO queue. Assuming that the input holiday is greater than the preset number of days (for example, the holiday is more than 5 days), the final holiday time series is: [Winter vacation: January 20, 2024 - February 5, 2024, Summer vacation: July 1, 2024 - August 31, 2024].

[0023] Record the time of each element in the driving school reservation sequence, and judge the driving school reservation sequence, When the number of new elements replaced reaches a preset number, a judgment is made based on the update time of the elements in the driving school reservation sequence. If there is an update time difference in the driving school reservation sequence that is greater than or equal to a preset time threshold, the replaced new element replaces the element with the earliest update time.

[0024] It should be noted that when using the LFU algorithm: for the students' application records, the LFU algorithm will give priority to retaining the test location with the most applications, and the resulting driving school appointment sequence is: application location A (10 times, the most recent time is October 1st), application location B (10 times, the most recent time is May 30th), application location C (8 times, the most recent time is September 30th), if the preset time threshold is 2 months, the preset number of times is 5 times, if the number of new application location D is 8 times, which exceeds the threshold, the time of application location B is greater than 2 months from the latest application location A, so application location D will replace application location B, and the final driving school appointment sequence is: application location A (10 times, the most recent time is October 1st), application location D (8 times, the most recent time is October 1st), application location C (8 times, the most recent time is September 30th).

[0025] S3: Allocate the appointment data according to the target person’s home location data and test location data; The target person's home location, the passing rate of the test location, and the test locations that have been tested are input into the matching degree formula to obtain the matching degree between the target person and each test location, and the target person's reservation data is allocated to the test location with the maximum matching degree; The matching degree formula is: ; In the formula, is the matching degree, is the first weight corresponding to the distance ratio, is the actual distance, is the maximum reference distance set, is the second weight corresponding to the pass rate, is the pass rate of the test location, is the third weight corresponding to the matching rate, is the number of matches of the completed test location in the driving school reservation sequence, Make reservations for all test locations in the sequence for the driving school.

[0026] It should be noted that, based on the calculated matching degree, the reservation data of the target person will be allocated to the test location with the highest matching degree. When the user is far away from all test locations, the weight will decrease, and vice versa. Secondly, when the pass rate of the test location is high, the test location with a high pass rate will be given priority for location selection. Finally, if the user has previously selected the test location according to the test location recommended by the driving school, the subsequent test location will meet the user's selection target, and the user's reservation data will be classified to the server of the corresponding test location, so that the data in the server has similarity, optimizing the server's compression level and improving storage efficiency.

[0027] S4: inputting the latest reservation data and historical examination information of the target person into the examination time point prediction model to obtain the predicted examination time point of the target person; Collect historical appointment data, convert the input data into time series format, and perform standardized preprocessing on the data. Use mean square error as the loss function, and finally obtain a trained test time point prediction model, wherein the test time point prediction model is trained using the Transformer model. The latest appointment data and historical test information of the target person are input into the test time point prediction model to obtain the predicted test time point of the target person, and then judge the predicted test time point of the target person.

[0028] It should be noted that all past appointment records of the target personnel are collected, including appointment time, test date, and location, and the collected historical data are converted into a time series format, where the time series format includes a timestamp and the corresponding feature value. The data is standardized by Z-score standardization or Min-Max scaling to eliminate the scale differences between different features; the mean square error is used as the loss function to evaluate the prediction performance of the model.

[0029] When the predicted examination time point of the target person is not within the latest holiday start time and end time with the same label in the holiday time series, or the historical appointment data and historical examination information are less than the preset quantity threshold, the middle value of the latest holiday start time and end time with the same label in the target person's holiday time series will be used as the predicted examination time point.

[0030] It should be noted that if there are not enough past appointment records of the target person, or the predicted test time is not within the latest holiday start and end time with the same label in the holiday time series, the middle value of the latest holiday start and end time with the same label in the target person's holiday time series will be used as the predicted test time; for example: the holiday time series is 7.15-9.1 for summer vacation and 7.8-9.1 for summer vacation. If the predicted test time is 7.10 for the test, the nearest holiday date is 7.15, and the test cannot be taken. At this time, the accuracy of the predicted test time will be low, and the middle value of the latest holiday start and end time with the same label will be used as the predicted test time.

[0031] S5: adjusting the priority of data storage according to the predicted test time point, the data storage priority formula and the storage priority adjustment formula; Use the data storage priority formula to adjust the historical reservation data, where the data storage priority formula is: ; in, The priority of data storage is is the current time, For the predicted test time point, A parameter for adjusting the rate at which priority changes.

[0032] Use the storage priority adjustment formula to adjust the priority of data storage to obtain the adjusted priority of data storage, where the storage priority adjustment formula is: ; in is the adjusted data storage priority. The priority of data storage is : is a function used to adjust the weight of storage priority according to the exam progress. is the adjustment coefficient, Indicates the time that has passed since registration.

[0033] It should be noted that, for example: when college students take a driver's license test, if the school is about to start when one of the subjects is completed, the students will return to school to study first and take the test again when there is enough vacation time. At this time, the data should be gradually put into cold storage for preservation, and the data that no longer needs to be frequently accessed should be transferred to cold storage to save storage costs; if the students are about to go on vacation, the study of the subject will be put back on the agenda, and attention will be paid to the relevant appointment data. Such data will be put into hot storage to reduce the delay in data access and improve the system response speed.

[0034] The driving license test must be completed within three years. If all subjects have not been completed in the third year, the target person will study the subjects in advance and pay attention to the relevant appointment data; therefore, the adjusted data storage priority is used for calculation, and the relevant data is extracted to the hot storage in advance, thereby completing the optimized storage of the data.

[0035] S6: Adjust the storage location of the target personnel data according to the adjusted data storage priority.

[0036] The historical appointment data of the target personnel whose data storage priority is greater than or equal to the preset priority threshold is extracted to the hot storage, and the historical appointment data of the target personnel whose data storage priority is less than the preset priority threshold is extracted to the cold storage.

[0037] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A method for optimizing and storing large-scale data of intelligent transportation, characterized by: The following steps are involved: S1: Collect historical data of target personnel and pre-process the historical data; S2: Use the FIFO algorithm and the LFU algorithm to obtain the target school's holiday time series and the driving school appointment sequence respectively; S3: Allocate the appointment data according to the target person’s home location data and test location data; S4: inputting the latest reservation data and historical examination information of the target person into the examination time point prediction model to obtain the predicted examination time point of the target person; S5: adjusting the priority of data storage according to the predicted test time point, the data storage priority formula and the storage priority adjustment formula; S6: Adjust the storage location of the target personnel data according to the adjusted data storage priority.

2. According to the method for optimizing and storing large-scale intelligent transportation data according to claim 1, it is characterized by: The collecting of historical data of the target person and preprocessing of the historical data include: obtaining the historical data of the target person, school holiday data and driving school reservation data, and performing data cleaning to fill in missing values, remove duplicate values ​​and remove abnormal values.

3. According to the method for optimizing and storing large-scale intelligent transportation data as claimed in claim 1, it is characterized by: The method uses the FIFO algorithm and the LFU algorithm to obtain the holiday time series and the driving school reservation sequence of the target school respectively, including: using the FIFO algorithm to input the holiday data of the school that is greater than a preset number of days each year to obtain the holiday time series of the target school, wherein the holiday time series includes the holiday start time and end time; The driving school reservation data at preset time intervals are inputted using the LFU algorithm to obtain a driving school reservation sequence, wherein the preset number of times and the update time of each element are set when the LFU algorithm is used.

4. A method for optimizing and storing large-scale intelligent transportation data according to claim 3, characterized in that: The LFU algorithm is used to input the driving school reservation data at a preset time interval to obtain a driving school reservation sequence, wherein a preset number of times is set when the LFU algorithm is used, including: recording the time of each element in the driving school reservation sequence, and judging the driving school reservation sequence, When the number of new elements replaced reaches a preset number, a judgment is made based on the update time of the elements in the driving school reservation sequence. If there is an update time difference in the driving school reservation sequence that is greater than or equal to a preset time threshold, the replaced new element replaces the element with the earliest update time.

5. According to the method for optimizing and storing large-scale intelligent transportation data as claimed in claim 1, it is characterized by: The step of allocating the reservation data according to the home location data and the test location data of the target person includes: inputting the home location of the target person, the pass rate of the test location, and the locations that have been tested into a matching degree formula to obtain the matching degree between the target person and each test location, and allocating the reservation data of the target person to the test location with the maximum matching degree; The matching degree formula is: ， In the formula, is the matching degree, is the first weight corresponding to the distance ratio, is the actual distance, is the maximum reference distance set, is the second weight corresponding to the pass rate, is the pass rate of the test location, is the third weight corresponding to the matching rate, is the number of matches of the completed test location in the driving school reservation sequence, Make reservations for all test locations in the sequence for the driving school.

6. The method for optimizing and storing large-scale intelligent transportation data according to claim 1 is characterized in that: The method of inputting the latest appointment data and historical examination information of the target person into the examination time point prediction model to obtain the predicted examination time point of the target person includes: collecting historical appointment data, converting the input data into a time series format, and performing standardized preprocessing on the data, using mean square error as a loss function, and finally obtaining a trained examination time point prediction model, wherein the examination time point prediction model is trained using a Transformer model, inputting the latest appointment data and historical examination information of the target person into the examination time point prediction model, obtaining the predicted examination time point of the target person, and judging the predicted examination time point of the target person.

7. A method for optimizing and storing large-scale intelligent transportation data according to claim 6, characterized in that: The latest reservation data and historical examination information of the target person are input into the examination time point prediction model to obtain the predicted examination time point of the target person, and the predicted examination time point of the target person is judged, including: when the predicted examination time point of the target person is not within the latest holiday start time and end time of the same label in the holiday time series, or the historical reservation data and historical examination information are less than a preset quantity threshold, the middle value of the latest holiday start time and end time in the holiday time series of the target person is used as the predicted examination time point.

8. The method for optimizing and storing large-scale intelligent transportation data according to claim 1 is characterized in that: The step of adjusting the data storage priority according to the predicted test time point, the data storage priority formula and the storage priority adjustment formula includes: adjusting the historical reservation data using the data storage priority formula, wherein the data storage priority formula is: ， in The priority of data storage is is the current time, For the predicted test time point, A parameter for adjusting the rate at which priority changes.

9. A method for optimizing and storing large-scale intelligent transportation data according to claim 8, characterized in that: The step of adjusting the data storage priority according to the predicted test time point, the data storage priority formula and the storage priority adjustment formula includes: adjusting the data storage priority using the storage priority adjustment formula to obtain the adjusted data storage priority, wherein the storage priority adjustment formula is: ， in is the adjusted data storage priority. The priority of data storage is : is a function used to adjust the weight of storage priority according to the exam progress, is the adjustment coefficient, Indicates the time that has passed since registration.

10. The method for optimizing and storing large-scale intelligent transportation data according to claim 1 is characterized by: The storage location of the target personnel data is adjusted through the adjusted data storage priority, including: extracting the historical appointment data of the target personnel whose data storage priority is greater than or equal to the preset priority threshold into hot storage, and extracting the historical appointment data of the target personnel whose data storage priority is less than the preset priority threshold into cold storage.

Images