Traffic accident visualization analysis method and system based on big data analysis technology

Abstract

The present invention discloses a traffic accident visualization analysis method and system based on big data analysis technology. The method comprises the following steps: collecting and collating provincial traffic accident and highway condition information from various websites, crawling provincial highway condition information through a Beautiful Soup library, and saving the data in a CSV file after unifying the data dimensions; processing and extracting features of the data in the CSV file to form a feature matrix and complete data proofreading; connecting to a MySQL database, uploading the data to the MySQL database, regularly crawling the data through a server and updating the database; performing mapping processing on the feature matrix, selecting appropriate features and inputting them into an SVM-Adaboost model for training, and selecting appropriate parameters through a grid search; compiling data from time and space dimensions, building a platform webpage using a Django framework, deploying the webpage to a server, and updating the platform data in real time; and through the construction of the platform, it is possible to obtain relevant information on domestic highway traffic accidents and predictions of related accidents in real time.

Description

Technical Field

[0001] The present invention relates to the field of data mining and artificial intelligence technology, and in particular to a traffic accident visualization analysis method and system based on big data analysis technology. Background Art

[0002] China's current expressways are booming, with mileage increasing year by year; in 2022, the expressway mileage increased by 8,200 kilometers. As of the end of 2022, my country has built 177,300 kilometers of expressways, of which 119,900 kilometers are national expressways; according to the latest released 2022 China Statistical Yearbook, the passenger volume of highways in 2021 reached 5 billion people, and the freight volume reached 39.1 billion tons.

[0003] In existing technologies, highway traffic accident statistics mainly rely on annual statistical reports and statistical yearbooks from relevant national authoritative websites. The statistical results are subject to certain data delays, statistical dimension limitations, and are difficult for citizens to access. Traffic news broadcasts rely on websites and radio, and such information sources cannot provide drivers with sufficient guidance and assistance. In addition, the characteristics of traffic accident data cannot be unified, and there is a lack of information integration, proofreading, and statistics. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a traffic accident visualization analysis method and system based on big data analysis technology.

[0005] According to a first embodiment of the present invention, a traffic accident visualization analysis method based on big data analysis technology includes the following steps:

[0006] Step S1: Collect and organize the traffic accident and highway condition information from the Times website of each province, crawl the highway condition information of each province through the BeautifulSoup library, unify the data dimensions and save it in a CSV file;

[0007] Step S2: Process and extract the features of the data in the CSV file, form a feature matrix and complete data proofreading;

[0008] Step S3: Connect to the MySQL database, upload data to the MySQL database, crawl data regularly through the server and update the database;

[0009] Step S4: Map the feature matrix, select appropriate features and input them into the SVM-Adaboost model training, output the prediction results and evaluate the model, select the appropriate SVM kernel function and select the appropriate parameters through grid search;

[0010] Step S5: Collect data from the time and space dimensions, build a platform webpage using the Django framework, deploy it to the server, and update the platform data in real time.

[0011] The specific process of step S4 is as follows: step S41: create the config.json parameter file and store the parameters required for the entire process of model building, training, and prediction; step S42: read the data frame type of the data set and map each column of the feature matrix; step S43: save the accident duration column as the label label, save other feature columns as the feature matrix feature_matrix, and standardize the feature matrix feature_matrix; step S44: calculate the variance of each column of the feature matrix feature_matrix, and calculate the feature matrix feature_matrix The covariance matrix after x and label label are combined; Step S45: According to the variance and covariance matrix of each column of the feature matrix feature_matrix, some features are discarded; Step S46: Call the SVM and AdaBoost models of the sklearn library, use the feature matrix feature_matrix as X and the label label as Y, divide the training set and the test set into 9:1, input the model for training, calculate the accuracy of the training set and the test set respectively; and visualize the accuracy of the training set and the test set; Step S47: Compare the performance of different models and select appropriate parameters through network search;

[0012] Among them, the Beautiful Soup library is used to parse HTML and XML files; CSV files are a commonly used data format; Mysql database is a data storage library; the sklearn library is a machine learning library in Python; SVM (Support Vector Machine) is a machine learning algorithm; AdaBoost is an integrated learning method.

[0013] According to the present invention, a traffic accident visualization analysis method based on big data analysis technology is used. The platform built by the above method can analyze large-scale traffic accident data and identify potential risk factors and high-risk areas for traffic accidents, thereby helping traffic management departments to take measures to improve road safety and reduce accident rates. Analyzing traffic accident data through the platform can help traffic planners better understand traffic flow patterns and the impact of accidents on traffic congestion, thereby helping to optimize road design and traffic management and reduce congestion problems. The real-time nature of the platform's webpage allows for real-time monitoring of traffic accidents and provides real-time information to emergency services when accidents occur, thereby facilitating rapid response to emergencies and improving rescue efficiency. The platform can predict possible future traffic accidents based on the data that has been mined, thereby helping traffic management departments to take preventive measures and reduce accident risks.

[0014] According to some embodiments of the present invention, the specific process of step S1 is as follows: step S11: collecting and organizing the real-time broadcast websites of traffic accidents and highway conditions in each province, organizing the websites and dividing them into data source websites and data proofreading websites; step S12: calling the Beautiful Soup library to crawl the data of the data source websites and the data proofreading websites respectively, and saving them in the main CSV file and the proofreading CSV file respectively; step S13: unifying the data information dimensions of each province, which should at least include the time of occurrence of the accident, recovery time, the city and road section where the accident occurred, detailed information about the accident, and the weather on that day. By collecting and organizing the real-time broadcast websites of traffic accidents and highway conditions in each province and crawling the data obtained by the crawler, the initial database required for building the platform is established.

[0015] According to some embodiments of the present invention, the specific process of step S2 is as follows: step S21: preliminary manual review of the data in the crawled csv file, and screening out unavailable data in the csv file; step S22: performing a first feature extraction on the screened data, and extracting it into three columns: accident information, accident process, and recovery time; step S23: performing a second feature extraction on the screened data, and extracting it into five columns: accident information, accident process, recovery time, road section, and accident details; step S24: crawling the city where the accident occurred through the Selenium library in Python; step S25: normalizing the data into ten columns of features: accident information, accident process, recovery time, road section, location, accident details, maximum temperature of the day, minimum temperature of the day, weather conditions of the day, and wind direction and wind level of the day; step S26: writing a Python script to compare the main csv file and the proofreading csv file one by one, and annotating the data in the main csv file according to the rules, and dividing the main csv file into data set_true csv file and data set_pseudo csv file according to the annotation results.

[0016] The rule in step S26 is as follows: if the data only appears in the main CSV file, it means that the accident only occurred on the data source website, and the authenticity of the accident needs to be considered, it is marked as 0; if it appears in both the main CSV file and the proofreading CSV file, and the accident recovery length calculated from the accidents on different websites deviates by no more than 10 minutes, it is marked as 1; if it appears in both the main CSV file and the proofreading CSV file, and the accident recovery length calculated from the accidents on different websites deviates by more than 10 minutes, it is marked as 2, and the accident broadcast time in the accident process column in the main CSV file is modified; the data marked as 1 and 2 in the main CSV file are stored in a new CSV file, recorded as Dataset_True CSV File, and the main CSV file is recorded as Dataset_Pseudo CSV File. By setting the above extraction rules, the initial data is subjected to multiple feature extractions, and the extracted features are classified into different files, completing the preprocessing process of collecting and crawling data in step S1.

[0017] According to some embodiments of the present invention, the specific process of step S3 is as follows: step S31: connecting to a MySQL database, uploading the dataset_true csv file, the dataset_pseudo csv file, and the proofreading csv file to the MySQL database; step S32: uploading all used Python scripts to the MySQL database, setting a regular crawling time according to the update speed of each website, and continuously updating the information in the dataset_true csv file, the dataset_pseudo csv file, and the proofreading csv file; step S33: regularly checking the three csv files in the server, manually checking whether there are outliers or abnormal crawling, and modifying and supplementing the crawler logic based on actual conditions. Through the database connection, the processed data is saved in the database, and the methods and tools for obtaining data are also uploaded to the database to facilitate regular data updates; manually checking the data content to observe whether the data is abnormal, and making modifications based on the cause of the abnormality.

[0018] According to some embodiments of the present invention, the specific process of step S5 is as follows: step S51: statistics on the data based on the same mapping rules as those used in model building; step S52: using the Django framework, connecting to the Mysql database, and building a platform webpage; step S53: deploying the platform webpage to the server, and updating the statistical data in real time; step S54: using html5, JavaScript, and CSS to edit and beautify the platform style and layout; the data statistical dimensions in step S51 include the following statistical dimensions: statistics on the number of accident vehicle types that occurred each month starting from this year, statistics on the number of accidents that occurred on provincial and national highways each month starting from this year, statistics on several highways with the most serious traffic accidents based on the average monthly number of accidents on each highway and the average recovery time of the accidents, statistics on the number of different types of traffic accidents that occurred each month starting from this year, statistics on several cities with the most traffic accidents in the country starting from this year, statistics on the proportion of severe weather and extreme weather in different types of accidents, and daily, weekly, and monthly statistics on the number of traffic accidents. Different statistical dimensions are used to collect statistics on data, and the statistical data is stored in the platform; Django is the web framework required for building the platform's web pages, and HTML5, JavaScript, and CSS are the core languages ​​for building web pages and network applications, which are used to define and structure the content of the web pages, define the interactive effects and animations of user operations, and control the layout style of the web pages; through the construction of the platform's web pages, real-time monitoring and management can be achieved.

[0019] According to some embodiments of the present invention, the specific process of the mapping in step S42 is to map the discrete text values ​​in the feature matrix to natural numbers such as 0, 1, 2, 3, and the continuous text values ​​to calculate the mean and variance values ​​of the data in the column of the feature matrix, take the normal distribution, take the confidence interval as 0.9, and those exceeding the confidence interval range are recorded as outliers. The confidence interval range is divided according to fixed intervals, and the division results are also mapped to natural numbers such as 0, 1, 2, 3.

[0020] According to a second aspect of an embodiment of the present invention, a traffic accident visualization analysis system based on big data analysis technology includes: a data storage module for storing all data in the system;

[0021] The data acquisition module collects and obtains traffic accident and highway condition information websites of various provinces, crawls the information on the websites and saves it in the data storage module;

[0022] The data processing module is used to process the data obtained by the data acquisition module and filter out the required feature data;

[0023] The model training module performs corresponding model training and selects appropriate parameters based on the data obtained by the data processing module;

[0024] The platform building module builds a fully functional web platform based on the output results of the model training module.

[0025] According to the present invention, a traffic accident visualization analysis system based on big data analysis technology can analyze large-scale traffic accident data and identify potential risk factors and high-risk areas for traffic accidents through the established traffic accident visualization system. The system can help traffic planners better understand traffic flow patterns and the impact of accidents on traffic congestion by analyzing traffic accident data; the system can also monitor traffic accidents in real time, provide real-time information to emergency services when accidents occur, and predict future accident rates based on historical data to reduce accident risks.

[0026] According to some embodiments of the present invention, the specific working principle of the model training module is to save the parameters required for the entire process of model construction, training, and prediction; map the feature data, divide the feature data into two categories: label label and feature matrix feature_matrix, and standardize the feature matrix feature_matrix; calculate the variance of each column of the feature matrix feature_matrix, and calculate the covariance matrix after the feature matrix feature_matrix and the label label are merged; discard some features according to the variance and covariance matrix of each column of the feature matrix feature_matrix; call the SVM-AdaBoost model in the system, use the feature matrix feature_matrix as X and the label label as Y, divide the training set and the test set into 9:1, input the model for training, and calculate the accuracy of the training set and the test set respectively; and visualize the accuracy of the training set and the test set; compare the performance of different models, select a suitable SVM kernel function and select suitable parameters through network search.

[0027] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is a flow chart of a traffic accident visualization analysis method based on big data analysis technology according to an embodiment of the present invention;

[0030] Figure 2 This is a flow chart of data acquisition in a traffic accident visualization analysis method based on big data analysis technology according to an embodiment of the present invention;

[0031] Figure 3 This is a flow chart of data preprocessing in a traffic accident visualization analysis method based on big data analysis technology according to an embodiment of the present invention;

[0032] Figure 4 This is a flowchart of data storage and updating in a traffic accident visualization analysis method based on big data analysis technology in an embodiment of the present invention.

[0033] Figure 5 This is a flowchart of the model construction in a traffic accident visualization analysis method based on big data analysis technology in an embodiment of the present invention.

[0034] Figure 6 This is a flow chart of a platform for building a traffic accident visualization analysis method based on big data analysis technology in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] It should be noted that when an element is referred to as being “fixed to” another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being “connected to” another element, it may be directly connected to the other element or there may be an intermediate element.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0038] Example 1

[0039] See Figures 1-6 As shown, a traffic accident visualization analysis method based on big data analysis technology includes the following steps:

[0040] Step S1: Collect and organize the traffic accident and highway condition information from the Times website of each province, crawl the highway condition information of each province through the BeautifulSoup library, unify the data dimensions and save it in a CSV file.

[0041] Furthermore, we collected and organized websites that report real-time traffic accidents and highway conditions in various provinces, such as the Highway Conditions Network (national traffic information), Jiangxi Travel Network, Jiangxi Highway Weibo Network, Hunan Highway Public Travel Service Network, Hunan Provincial Department of Transportation, Sichuan Provincial Department of Transportation, etc.; sorted and classified each website into two categories: data source websites and data proofreading websites. The classification standard is determined by the accident information dimension provided by the website. The data source website should at least include the time of occurrence of the accident, the recovery time, the city and road section where the accident occurred, and detailed information about the accident; websites that do not meet the minimum conditions are classified as data proofreading websites. Different crawler logic is written for different website source codes. Through the python crawler script, the Beautiful Soup library is called to crawl the data of the data source website and the data proofreading website respectively. The data is saved in the main csv file and the proofreading csv file respectively.

[0042] Furthermore, to prevent web scraping from being blocked, the script sets multiple IP_Agent and User_Agent. When a crawl is blocked, it randomly changes the proxy information to continue crawling. IP_Agent refers to the server proxy, which allows the client to send requests to other servers through the proxy server during crawling operations. User_Agent refers to the service proxy used to request access to the browser. After receiving the request, the server will use it to determine the browser, operating system, or other environmental information used by the visitor.

[0043] Step S2: Process and extract the features of the data in the CSV file, form a feature matrix and complete data proofreading.

[0044] Furthermore, the data in the crawled csv files are manually reviewed initially to filter out unusable data in the csv files; the filtered data is subjected to the first feature extraction, and extracted into three columns: accident information, accident process, and recovery time. The accident information only contains a detailed text description of the accident; the accident process is a multi-element double-layer nested list, in which each element is a binary list containing the broadcast time and the corresponding broadcast information; the time of the first broadcast and the last broadcast (generally the broadcast when the accident occurred and the broadcast when the accident was recovered) are subtracted to obtain the third column, the accident recovery time. The date calculation is implemented through the datetime library in python; among them, datetime provides various types of functions for performing date and practical operations;

[0045] Furthermore, the filtered data is subjected to a second feature extraction, and the five columns extracted are accident information, accident process, recovery time, road section, and accident details; among them, the accident process is a binary double-layer nested list, and the redundant reports that are not accidents and not accidents recovered are deleted on the basis of the previous ones, leaving only two binary lists containing the broadcast time and the corresponding broadcast information; the road section is the highway number, highway name, and specific mileage, where the highway number and specific mileage are both a combination of single letters and numbers; the accident details are long text data containing information such as the type of accident, the number of vehicles affected by the accident, and the model of the vehicles.

[0046] Going a step further, a Python crawler script was written to determine the specific city where the accident occurred based on the highway number and its specific kilometer number. Conventional crawling methods cannot directly obtain this information from the website, so the solution is Selenium crawler. Selenium is a popular automated testing tool. The Selenium library is used to simulate browser clicks, automatically searching for highways and specific kilometer numbers in the browser, and then extracting the city from the search results. The data in each column is then normalized, including removing leading and trailing spaces, removing meaningless alphanumeric characters, and modifying and standardizing data types.

[0047] Furthermore, a python script is written to compare the main CSV file and the proofreading CSV file one by one, and the data in the main CSV file is annotated according to the following rules;

[0048] Labeling rules: If data appears only in the main CSV file, it indicates that the incident only occurred on the data source website, and the authenticity of the incident is questionable, so it is labeled 0. If it appears in both the main CSV file and the proofreading CSV file, and the calculated incident recovery lengths for incidents on different websites do not deviate by more than 10 minutes, it is labeled 1. If it appears in both the main CSV file and the proofreading CSV file, and the calculated incident recovery lengths for incidents on different websites deviate by more than 10 minutes, it is labeled 2, and the incident broadcast time in the incident process column of the main CSV file is modified. The data marked 1 and 2 in the main CSV file are stored in a new CSV file, labeled Dataset_True CSV File, and the main CSV file is labeled Dataset_Fake CSV File. By setting the above extraction rules, multiple feature extractions are performed on the initial data, and the extracted features are classified into different files.

[0049] Step S3: Connect to the MySQL database, upload the data to the MySQL database, crawl the data regularly through the server and update the database.

[0050] Furthermore, connect to the MySQL database and upload the dataset_true csv file, dataset_pseudo csv file, and proofreading csv file to the MySQL database; upload all the Python scripts used to the MySQL database, set a regular crawling time according to the update speed of each website, and continuously update the information in the dataset_true csv file, dataset_pseudo csv file, and proofreading csv file;

[0051] Furthermore, we regularly check the three CSV files in the server, manually check whether there are any abnormal values ​​or abnormal crawling, and modify and supplement the crawler logic based on the actual situation.

[0052] Step S4: Map the feature matrix, select appropriate features and input them into the SVM-Adaboost model training, output the prediction results and evaluate the model, select the appropriate SVM kernel function and select the appropriate parameters through grid search.

[0053] Furthermore, create a config.json parameter file to store the parameters required for the entire process of model building, training, and prediction; read the data frame type of the data set and map each column of the feature matrix; save the accident duration column as the label label, and save other feature columns as the feature matrix feature_matrix, and standardize the feature matrix feature_matrix; calculate the variance of each column of the feature matrix feature_matrix, and calculate the covariance matrix after merging the feature matrix feature_matrix and the label label; discard some features based on the variance and covariance matrix of each column of the feature matrix feature_matrix; call the SVM and AdaBoost models of the sklearn library, use the feature matrix feature_matrix as X and the label label as Y, divide the training set and test set into 9:1, input the model for training, and calculate the accuracy of the training set and test set respectively; and visualize the accuracy of the training set and test set; compare the performance of different models, select the appropriate SVM kernel function and select the appropriate parameters through network search.

[0054] The specific process is: 1. Record the training data set of feature_matrix as:

[0055] T={(X1,y1),(X2,y2),...(X N ,y N )}

[0056] Where N is the number of samples, Represents the ten features of the sample, y n Is a floating point number representing the true value of the nth sample, It is the instance space.

[0057] 2. Initialize the weight distribution of the training data T. Each sample is initialized with an equal weight, that is, each sample has the same probability of being selected.

[0058]

[0059] D n Represents the weight distribution in the nth cycle, w ni Represents the weight of the i-th sample in the n-th cycle.

[0060] 3. In the first iteration, N samples are sampled with replacement from the existing dataset X according to D1 to form a new training set X t Since the weight distribution of D1 is equal everywhere, all samples have the same probability of being drawn in the first iteration.

[0061] 4. In the training set X t Train and build a regression tree according to the Cart tree rule.

[0062] The Cart tree rule refers to the Classification and Regression Tree, which generates the largest possible decision tree based on the training data set, and uses the validation data set to prune the generated tree and select the optimal subtree.

[0063] 5. X t All samples on the input regression tree to obtain the prediction result is recorded as f(x i ), the true value of the sample is y i , where i = 1, 2, ..., N.

[0064] 6. Calculate the loss function on a single sample i:

[0065] L i =Loss(f(x i ), y i ).

[0066] 7. Calculate the weighted average loss on all samples And the confidence β of the current algorithm:

[0067]

[0068]

[0069] Here, λ is a constant that prevents the denominator from being zero.

[0070] It is easy to see that when the weighted average loss is high, the confidence β is small; when the weighted average loss is low, the confidence β is large.

[0071] 8. Update sample weight w i And the current (i.e. first iteration) regression tree weight φ1:

[0072]

[0073] φ1=-logβ

[0074] Continuing from step 7, when the weighted average loss is high, w i The larger it is, the smaller the regression tree weight φ is.

[0075] 9. Repeat steps 3 to 8, updating D in each iteration n The weight value in , and record the weight φ of the regression tree until the iteration ends, at which point the training is complete. The final output value of the algorithm is:

[0076]

[0077] SVM is a machine learning method that can be used for both linear and nonlinear data. It maps training data to a higher dimension, searches for the optimal linear hyperplane in the new dimension, and ultimately determines the sample's category based on this hyperplane. The Adaboost algorithm combines several weak regressors into a strong classifier. Through iteration, it adjusts the sample weights after each round of training and adaptively changes the distribution of training samples based on these adjusted weights, allowing the weak regressors to focus more on samples with high prediction difficulty, thereby improving prediction accuracy. By combining the Adaboost algorithm with the SVM weak classifier, the classification results of each layer can be weighted and combined, further improving the accuracy of the prediction algorithm.

[0078] Furthermore, the specific mapping process involves mapping discrete text values ​​in the feature matrix to natural numbers such as 0, 1, 2, and 3. For continuous text values, the mean and variance of the data in that column of the feature matrix are calculated, using a normal distribution with a confidence interval of 0.9. Data outside the confidence interval are classified as outliers. Within the confidence interval, data is divided into fixed intervals, and the division results are also mapped to natural numbers such as 0, 1, 2, and 3. Step S5: Statistical data is collected from the time and space dimensions. The platform webpage is built using the Django framework and deployed to the server for real-time updates.

[0079] Furthermore, based on the same mapping rules as the model building, the data is statistically analyzed; the Django framework is used to connect to the Mysql database to build the platform web page; HTML5, JavaScript and CSS are used to edit and beautify the platform style and layout.

[0080] Furthermore, the data statistical dimensions include the following statistical dimensions: statistics on the number of types of vehicles involved in accidents each month starting from this year, statistics on the number of accidents occurring on provincial and national highways each month starting from this year, statistics on several highways with the most serious traffic accidents based on the average monthly number of accidents on each highway and the average recovery time of accidents, statistics on the number of different types of traffic accidents occurring each month starting from this year, statistics on several cities with the most traffic accidents nationwide starting from this year, statistics on the proportion of severe weather and extreme weather in different types of accidents, and statistics on the daily, weekly and monthly year-on-year growth in the number of traffic accidents.

[0081] Beneficial effects of this embodiment:

[0082] 1. Improve traffic safety: The platform can analyze large-scale traffic accident data and identify potential risk factors and high-risk areas for traffic accidents. This helps traffic management departments take measures to improve road safety and reduce accident rates.

[0083] 2. Reduce traffic congestion: By analyzing traffic accident data, the platform can help traffic planners better understand traffic flow patterns and the impact of accidents on traffic congestion; it can help optimize road design and traffic management, reducing congestion problems.

[0084] 3. Rapid response to emergencies: The platform can monitor traffic accidents in real time and provide real-time information to emergency services when an accident occurs; this helps to quickly respond to emergencies and improve rescue efficiency.

[0085] 4. Accident prediction: Based on historical data and data mining technology, the platform can predict traffic accidents that may occur in the future; this helps traffic management departments take preventive measures and reduce accident risks.

[0086] 5. Decision Support: The platform provides visualization tools and data analysis capabilities to help government decision makers better understand traffic accident trends and patterns; thus, it helps to formulate more effective policies and plans.

[0087] 6. Public participation: The platform can be open to the public, allowing citizens to better understand traffic accident data and safety issues; it can improve the public's safety awareness and encourage them to drive more carefully.

[0088] 7. Cost savings: By better managing traffic accidents and congestion, the platform helps save on road maintenance and rescue costs.

[0089] Example 2

[0090] A traffic accident visualization analysis system based on big data analysis technology includes: a data storage module for storing all data in the system;

[0091] The data acquisition module collects and obtains traffic accident and highway condition information websites of various provinces, crawls the information on the websites and saves it in the data storage module;

[0092] The data processing module is used to process the data obtained by the data acquisition module and filter out the required feature data;

[0093] The model training module performs corresponding model training and selects appropriate parameters based on the data obtained by the data processing module;

[0094] The platform building module builds a fully functional web platform based on the output results of the model training module.

[0095] Furthermore, the specific working principle of the above-mentioned model training module is to save the parameters required for the entire process of model construction, training, and prediction; map the feature data, divide the feature data into two categories: label label and feature matrix feature_matrix, and standardize the feature matrix feature_matrix; calculate the variance of each column of the feature matrix feature_matrix, and calculate the covariance matrix after the feature matrix feature_matrix and label label are combined; discard some features according to the variance and covariance matrix of each column of the feature matrix feature_matrix; call the SVM and AdaBoost models in the system, use the feature matrix feature_matrix as X and the label label as Y, divide the training set and test set into 9:1, input the model for training, calculate the accuracy of the training set and test set respectively; and visualize the accuracy of the training set and test set; compare the performance of different models, select the appropriate SVM kernel function and select the appropriate parameters through network search.

[0096] Beneficial effects of this embodiment:

[0097] 1. Through the traffic accident visualization system built, the system can analyze large-scale traffic accident data and identify potential risk factors and high-risk areas for traffic accidents. By analyzing traffic accident data, the system can help traffic planners better understand traffic flow patterns and the impact of accidents on traffic congestion; the system can also monitor traffic accidents in real time and provide real-time information to emergency services when accidents occur. It can also predict future accident rates based on historical data to reduce accident risks.

[0098] 2. Integrate the processing modules with different functions to facilitate subsequent use and troubleshooting.

[0099] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as a limitation to the invention.

[0100] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "example," "specific example," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0101] Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present embodiment application. The appearance of this phrase in various positions in the specification does not necessarily mean that they are all the same embodiments, nor are they independent or alternative embodiments that are mutually exclusive with other embodiments. It can be understood explicitly and implicitly by those skilled in the art that the embodiments described herein can be combined with other embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0102] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A traffic accident visualization analysis method based on big data analysis technology, characterized in that: The following steps are included: Step S1: Collect and organize the provincial traffic accident and highway condition information from the Times website, crawl the provincial highway condition information using the Beautiful Soup library, unify the data dimensions and save it in a CSV file; Step S2: Process and extract the features of the data in the CSV file, form a feature matrix and complete data proofreading; The method includes the following steps: step S21: performing a preliminary manual review of the data in the crawled CSV file and filtering out unusable data in the CSV file; step S22: performing a first feature extraction on the filtered data, extracting the data into three columns: accident information, accident process, and recovery time; step S23: performing a second feature extraction on the filtered data, extracting the data into five columns: accident information, accident process, recovery time, road section, and accident details; step S24: crawling the city where the accident occurred through the Selenium library in Python; step S25: normalizing the data into ten columns of features: accident information, accident process, recovery time, road section, location, accident details, maximum temperature of the day, minimum temperature of the day, weather conditions of the day, and wind direction and wind level of the day; step S26: writing a Python script to compare the main CSV file and the proofreading CSV file one by one, and annotating the data in the main CSV file according to the rules, and dividing the main CSV file into a data set_true CSV file and a data set_pseudo CSV file according to the annotation results; Step S3: Connect to the MySQL database, upload data to the MySQL database, crawl data regularly through the server and update the database; Step S4: Map the feature matrix, select appropriate features and input them into the SVM-Adaboost model training, output the prediction results and evaluate the model, select the appropriate SVM kernel function and select the appropriate parameters through grid search; Step S5: Collect data from time and space dimensions, build a platform webpage using the Django framework, deploy it to the server, and update the platform data in real time; Among them, the specific process of step S4 is, step S41: create a config.json parameter file, and store the parameters required for the entire process of model construction, training, and prediction; step S42: read the data frame type of the data set, and map each column of the feature matrix. The specific process of the mapping is to map the discrete text values ​​in the feature matrix to natural numbers, and for the continuous text values, calculate the mean and variance values ​​of the data in the feature matrix of this column, take a normal distribution, and take a confidence interval of 0.

9. Those exceeding the confidence interval range are recorded as outliers. The confidence interval range is divided according to fixed intervals, and the division results are also mapped to natural numbers; step S43: save the accident duration column as the label label, save other feature columns as the feature matrix feature_matrix, and standardize the feature matrix feature_matrix Processing; Step S44: Calculate the variance of each column of the feature matrix feature_matrix, and calculate the covariance matrix after the feature matrix feature_matrix and the label label are merged; Step S45: According to the variance and covariance matrix of each column of the feature matrix feature_matrix, discard some features; Step S46: Call the SVM and AdaBoost models of the sklearn library, use the feature matrix feature_matrix as X, the label label as Y, divide the training set and the test set into 9:1, input the model for training, and calculate the accuracy of the training set and the test set respectively; and visualize the accuracy of the training set and the test set; Step S47: Compare the performance of different models, select a suitable SVM kernel function and select suitable parameters through network search; Among them, the Beautiful Soup library is used to parse HTML and XML files; CSV files are a commonly used data format; Mysql database is a data storage library; the sklearn library is a machine learning library in Python; SVM is a machine learning algorithm; the kernel function is a function in the SVM algorithm used to convert input data into a higher-dimensional space; AdaBoost is an integrated learning method.

2. The traffic accident visualization analysis method based on big data analysis technology according to claim 1 is characterized in that: The specific process of step S1 is: Step S11: Collect and organize the real-time broadcast websites of traffic accidents and highway conditions in each province, organize the websites and divide them into data source websites and data proofreading websites; Step S12: Call the Beautiful Soup library to crawl data from the data source websites and data proofreading websites respectively, and save them in the main CSV file and the proofreading CSV file respectively; Step S13: Unify the data information dimensions of each province, which should at least include the time of occurrence of the accident, recovery time, the city and road section where the accident occurred, detailed information of the accident, and the weather on that day.

3. The traffic accident visualization analysis method based on big data analysis technology according to claim 2 is characterized in that: The rule in step S26 is: if the data only appears in the main CSV file, it means that the accident only occurred on the data source website, and the authenticity of the accident needs to be considered, it is marked as 0; if it appears in both the main CSV file and the proofreading CSV file, and the deviation of the accident recovery length calculated for accidents on different websites does not exceed 10 minutes, it is marked as 1; if it appears in both the main CSV file and the proofreading CSV file, and the deviation of the accident recovery length calculated for accidents on different websites exceeds 10 minutes, it is marked as 2, and the accident broadcast time in the accident process column of the main CSV file is modified; Save the data marked as 1 and 2 from the main CSV file into a new CSV file, record it as dataset_true CSV file, and record the main CSV file as dataset_pseudo CSV file.

4. The traffic accident visualization analysis method based on big data analysis technology according to claim 1 is characterized in that: The specific process of step S3 is: Step S31: Connect to the Mysql database and upload the dataset_true csv file, dataset_pseudo csv file and proofreading csv file to the Mysql database; Step S32: Upload all Python scripts used to the Mysql database, set regular crawling time according to the update speed of each website, and continuously update the information in the dataset_true csv file, dataset_pseudo csv file and proofreading csv file; Step S33: Regularly check the three csv files in the server, manually check whether there are abnormal values ​​or abnormal crawling, and modify and supplement the crawler logic based on actual conditions.

5. The traffic accident visualization analysis method based on big data analysis technology according to claim 1 is characterized in that: The specific process of step S5 is: Step S51: Based on the same mapping rules as the model building, statistics are performed on the data; Step S52: Use the Django framework to connect to the Mysql database and build a platform webpage; Step S53: Deploy the platform webpage to the server and update the statistical data in real time; Step S54: Use HTML5, JavaScript and CSS to edit and beautify the platform style and layout.

6. The traffic accident visualization analysis method based on big data analysis technology according to claim 5 is characterized in that: The data statistical dimensions in step S51 include the following statistical dimensions: statistics on the number of accident vehicles involved in each month since this year, statistics on the number of accidents involved in provincial and national highways each month since this year, statistics on several highways with the most serious traffic accidents based on the average monthly number of accidents on each highway and the average recovery time of accidents, statistics on the number of different types of traffic accidents involved in each month since this year, statistics on several cities with the most traffic accidents in the country since this year, statistics on the proportion of severe weather and extreme weather in different types of accidents, and statistics on the daily, weekly and monthly year-on-year increases in the number of traffic accidents.

7. A traffic accident visualization analysis system based on big data analysis technology, referring to the method according to any one of claims 1 to 6, characterized in that: include: Data storage module, used to store all data in the system; The data acquisition module collects and obtains traffic accident and highway condition information websites of various provinces, crawls the information on the websites and saves it in the data storage module; The data processing module is used to process the data obtained by the data acquisition module and filter out the required feature data; The model training module performs corresponding model training and selects appropriate parameters based on the data obtained by the data processing module; The platform building module builds a fully functional web platform based on the output results of the model training module.

8. The traffic accident visualization analysis system based on big data analysis technology according to claim 7 is characterized in that: The specific working principle of the model training module is to save the parameters required for the entire process of model construction, training, and prediction; perform mapping processing on the feature data, divide the feature data into two categories: label label and feature matrix feature_matrix, and perform standardization on the feature matrix feature_matrix; calculate the variance of each column of the feature matrix feature_matrix, and calculate the covariance matrix after the feature matrix feature_matrix and label label are combined; according to the variance and covariance matrix of each column of the feature matrix feature_matrix, discard some features; Call the AdaBoost model in the system, use the feature matrix feature_matrix as X and the label label as Y, divide the training set and test set into 9:1, input the model for training, calculate the accuracy of the training set and test set respectively; and visualize the accuracy of the training set and test set; compare the performance of different models, select the appropriate SVM kernel function and select the appropriate parameters through network search.

Images