Method and system for monitoring and predicting visitors in garden based on multi-modal data analysis

Abstract

The invention relates to the technical field of data analysis and prediction, in particular to an in-park tourist monitoring and predicting method and system based on multi-modal data analysis, and discloses a scenic spot in-park tourist number real-time monitoring and predicting method based on fusion of different-modal data analysis and multi-model integration. According to the invention, on the basis of a neural network technology, multi-modal data such as signaling data, traffic checkpoints, video monitoring, scenic area wifi-hotspot and weather data are taken as basic data, and a multi-model fusion prediction technology is combined, so that real-time in-park passenger flow detection and prediction of the scenic area are realized. The method integrates more passenger flow influence factors and the like, is more accurate and scientific than a traditional data superposition calculation method, is low in implementation cost, and has high generalization performance.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis and prediction, and specifically to a method and system for monitoring and predicting the number of tourists in a scenic area based on multimodal data analysis, and a method for monitoring and predicting the number of tourists in a scenic area in real time based on the integration of different modal data analysis and multi-model integration. Background Art

[0002] The continuous growth of domestic public tourism demand has kept the number of tourists in tourist destinations at a high level. While the surge in passenger flow has brought economic growth, it has also brought a series of challenges. The surge in passenger flow has brought tremendous pressure on scenic spot facilities and seriously affected tourists' perception of tourism. Accurate statistics of tourist passenger flow can improve the management capacity of scenic spots, reduce the negative impact of the surge in passenger flow, and ensure the sustainable and healthy development of scenic spots. At present, tourist destinations have made a lot of investment in the infrastructure of tourism public services, but they still cannot accurately perceive the passenger flow and passenger flow density of scenic spots. At present, the real-time in-park data of scenic spots obtained by statistics from a single data source for large scenic spots and open scenic spots is often very different from the actual situation. Due to the lack of data accumulation and big data analysis capabilities of a single data source, it is impossible to explore the travel patterns of tourists. The real-time in-park tourist data obtained by combining and predicting multiple simple data sources is more in line with the actual situation and the needs of data analysis. Therefore, it is very necessary to combine multiple data sources and passenger flow influencing factors to predict and analyze in-park tourists. This invention analyzes tourist behavior and makes real-time predictions and analyses of tourists in the park through a combination of single data sources such as mobile phone signaling data, traffic checkpoints, video surveillance, scenic spot wifi hotspots, and weather data, providing data support for the standardization of tourism market management and intelligent guidance of tourist travel. Summary of the invention

[0003] The present invention combines single data source statistical data to obtain real-time in-park tourist data of scenic spots through multi-model fusion. Compared with the traditional in-park tourist data statistics through single data source, it has higher accuracy and real-time performance, and can objectively and truly reflect the current actual situation of the scenic spot without human interference. It effectively solves the problem of passenger flow statistics in large scenic spots and developed scenic spots, and improves the industry management efficiency and public service level of tourism management departments.

[0004] The method for monitoring and predicting tourists in the park based on multimodal data analysis includes the following steps: S1, multimodal data extraction and preprocessing; use signaling data to obtain the number of users accessing the operator base stations around the scenic area, use equipment with pattern recognition algorithms to obtain information such as the vehicle models and numbers at the traffic checkpoints around the scenic area, connect to the surveillance video of the key areas of the scenic area, obtain the number of tourists in the key areas of the scenic area through the video analysis algorithm, obtain the scenic area wifi data through the interface, obtain the number of tourists accessing the scenic area WIFI system, extract the real-time weather information of the scenic area environmental detection equipment that meets the GB 3095-2018 standard or the authoritative department, and obtain the current environment and weather conditions of the scenic area; S2, multimodal feature extraction and feature vector fusion Through the multi-modal data such as scenic spot signaling data, traffic checkpoints, video surveillance, scenic spot wifi hotspots, weather data, etc. provided by the operator, the maximum tourist carrying capacity of the scenic spot is used as the maximum value of the data and 0 is used as the minimum value. The noise, missing values ​​and outliers of the signaling data, traffic checkpoints, video surveillance, scenic spot wifi hotspot data, and weather data are preprocessed. Finally, the data is Z-score standardized. Z-score standardization is a common data processing method. The data is subtracted from its mean by attribute and divided by its standard deviation. The formula is: in is the standardized value, is the original data, is the mean, is the standard deviation; the data of each attribute is clustered around 0, and the variance is 1; the eigenvalues ​​after data standardization are used as their feature subvectors. Finally, they are concatenated in the order of {time, signaling data feature subvector, in-park vehicle feature subvector, wifi data feature subvector, video surveillance feature subvector, weather feature subvector} to form a complete feature vector; S3, multi-model stacking integration The establishment of the model is divided into two parts: sub-model establishment and multi-model integration; the characteristics describing the real-time status of the scenic spot are input into the two sub-models, the decision tree model based on the gradient boosting algorithm and the deep neural network model, respectively, and an integrated model is established based on the prediction results of the two sub-models. Finally, the integrated model is used to calculate the real-time passenger flow of the scenic spot.

[0005] The model building method is as follows: (1) Establish a gradient boosting decision tree model Use LightGBM to build a gradient boosting decision tree model based on the distributed gradient boosting framework of the decision tree algorithm Input: Sample set , , where D is the sample collection time, n is the total number of samples; Output: Gradient Boosted Decision Tree Model Initialize decision number ; Set the total number of decision trees trained by the gradient boosting decision tree model to M, and calculate the residual of each sample for m = 1, 2, ..., M respectively; then the residual of the sample is defined as ; , i=1, 2, ..., n; n is the total number of samples, and the residuals of each sample are fitted by using the regression decision tree model ; Learn and get the mth decision tree ,in is the parameter in the mth decision tree; Get the gradient boosting decision tree model , the output is the real-time number of tourists in the scenic area predicted by the gradient boosting decision tree model; (2) Establish a deep neural network model: Use the Keras framework to build a long short-term memory (LSTM) neural network model; standardize the signaling data, in-park vehicle data, WiFi data, video surveillance data, and weather data according to all the numerical data types in the feature data. When building the model, the batch dimension does not need to be displayed and is set to None to ensure that the input layer has the same shape as the training data set, that is, the shape of the input layer is [None, 14, 6], and then the dimension is adjusted to [None, 14, 6, 1] through reshaping so that the subsequent two-dimensional convolution operation can be performed; set the number of parameter filters, the size of the convolution kernel, and the regularization for the convolution layer, and set Padding=same to fill the input with zero values. Using the BN layer to standardize the batch can speed up the training of the model, alleviate the disappearance of the gradient and obey the normal distribution; adding a RELU excitation layer after batch normalization makes it easier to find the gradient and improves the convergence speed, and then using pooling to further amplify the feature information. At this time, after the convolution network is completed, the spatial relationship between the features is extracted; after the convolution is completed, the convolution channel is adjusted 1*1; after the convolution process is completed, first reduce one dimension through the squeeze layer, so that the shape changes from four dimensions to three dimensions, and then use LSTM, where the first layer processes the input sequence and returns the output sequence of each time step, and the second layer processes the output sequence of the first layer and returns a single output of the entire sequence to process the time series information of the data. Then add a Dense layer containing 64 neurons to perform nonlinear transformation on the input data and learn the nonlinear relationship between the input features and the output. Finally, use the fully connected layer as the output layer; A multi-model ensemble model is built based on the prediction results of the two sub-models: a decision tree regression model is created using the scikit-learn library; for all samples in the training set , ,Will Input the trained gradient boosting decision tree model separately Get the prediction results And the deep neural network model g(T) to get the prediction results , construct a new training sample set for the integrated model , in the input space where the new sample set is located, the least squares method is used to segment the variables and Traverse and calculate the loss value of each segmentation point, select the segmentation variable and segmentation point with the smallest loss value, recursively divide each area into two sub-areas and determine the output value of each sub-area, and build a regression decision tree model. The output of the model is the predicted real-time number of tourists in the scenic area.

[0006] The method for monitoring and predicting tourists in the park based on multimodal data analysis, the S2 multimodal data analysis, feature extraction and feature fusion include: b1. Collect the number of tourists calculated based on signaling data provided by the scenic spot operator and extract the signaling data features; Where n represents the number of people accessing the nth base station. The data is normalized by Z-score to form the signaling data feature subvector ; b2. Analyze the traffic checkpoint data and extract the characteristics of vehicles in the park: extract the information of vehicles and models entering and leaving the scenic area through the traffic checkpoints, and remove the vehicles leaving from the data of vehicles entering; calculate the number of vehicles and models in the park, and then calculate the corresponding coefficients of models and number of people. Estimated number of people , where n represents the model, represents the number of vehicle models entering the area. represents the number of the i-th vehicle type leaving the area. After Z-score normalization of the data, the characteristic sub-vector of vehicles in the park is formed. ; b3. Analyze weather data and extract weather characteristics: divide weather into different levels according to weather conditions, and after Z-score standardization of the data, form a weather characteristic subvector ; b4. Analyze wifi hotspot data and extract wifi data features: obtain the number of online wifi terminals in all scenic spots and calculate the average of these data. After Z-score normalization of the data, the characteristic sub-vector of WiFi data is formed ; b5. Analyze video surveillance data and extract video surveillance data features: obtain real-time passenger flow data from passenger flow counting cameras in key areas of all scenic spots, and sum these data ,in Represents the number of tourists captured by the passenger flow camera. After the data is normalized by Z-score, it constitutes the feature subvector of the video surveillance data ; b6. Combine the signaling data feature sub-vector, the in-park vehicle feature sub-vector, the WiFi data feature sub-vector with the analyzed video surveillance data, and compare it with the real-time number of in-park tourists in the scenic spot corresponding to the itinerary The complete feature vector describing the number of tourists in the scenic spot in real time is composed as follows: .

[0007] A system for monitoring and predicting methods of tourists in the park based on multimodal data analysis, wherein the system is a deep neural network model; Part 1: Input layer: standardize the signaling data, in-park vehicle data, WiFi data, video surveillance data, and weather data according to all the numerical data types in the feature data to serve as the input data for the model input layer; Part 2: Reshape to adjust the dimension: The shape of the input layer is [None, 10, 6], and the dimension is adjusted to [None, 10, 6, 1] through reshape; Part 3: BN+Relu convolution; the number of output channels of the convolution layer is 128, the size of the convolution kernel is 3x3 and L2 regularization, and Padding=same is set to fill the input with 0 values; L2 regularization is applied, and the regularization coefficient is 0.01. Using the BN layer to standardize the batch can speed up the training of the model, alleviate the disappearance of the gradient and obey the normal distribution; after batch normalization, a RELU excitation layer is added, and the processed data output is [None, 14, 6, 128]; Part 4: Pooling sampling. The size of the pooling window is 2x2. Pooling further amplifies the feature information. The output data of the pooling process is [None, 14, 6, 128].

[0008] Part 5: Convolution adjusts the number of channels. The number of output channels of the convolution layer is 1, the size of the convolution kernel is 3x3 and L2 regularization is used, the regularization coefficient is 0.01, and the padding strategy Padding=same is set to fill the input with 0 values. The output data is [None, 14, 6, 1]; Part 6: squeeze layer, use the squeeze layer to remove the last dimension, adjust the dimension of the data from four dimensions to three dimensions, and the output data is [None, 14, 4]; Part 7: The first LSTM layer uses a bidirectional LSTM layer with 128 units, uses ReLU as the activation function, and applies L2 regularization with a coefficient of 0.01; Part 8: The second LSTM layer uses a bidirectional LSTM layer with 64 units, uses ReLU as the activation function, applies L2 regularization, and the coefficient is 0.01; the output data is [None, 128]; Part 9: Fully connected layer, using a fully connected layer with an output dimension of 64, using ReLU as the activation function, and the output data is [None, 64]; Part 10: Output layer, a fully connected layer with an output dimension of 1 and an activation function of sigmoid. As the output layer of the network, the real-time number of tourists in the scenic area is predicted using a deep neural network model.

[0009] Beneficial effects of the present invention: First, compared with the existing prediction methods for tourists in the park, this method solves the problem of difficulty and low accuracy in detecting and predicting tourists in the park using a single data collection in open areas such as large scenic spots, open scenic spots, and urban squares; Second, traditional tourist prediction generally uses a single data source for calculation, which is heavily dependent on a single data source and cannot integrate multiple influencing factors such as weather and traffic into the algorithm; the obtained scenic spot passenger flow data itself is inaccurate; there is no other data for verification and correction in the subsequent data output process, and the traditional method generally uses a simple parameter correction method, relying on subjective experience; resulting in a large difference between the prediction results and the actual situation; Based on neural network technology, the present invention uses multi-modal data such as signaling data, traffic checkpoints, video surveillance, scenic spot wifi-hotspots, weather data, etc. as basic data, combined with multi-model fusion prediction technology, to realize the detection and prediction of real-time in-park passenger flow in scenic spots. It incorporates more passenger flow influencing factors, etc., which is more accurate and scientific than the traditional data superposition calculation method, and has low implementation cost and strong promotion.

[0010] Possible future application scenarios of the present invention include: 1. Scenic area management: Provide real-time crowd flow detection and effective prediction for scenic spots, help scenic area management departments better understand tourist flow, arrange resources reasonably, optimize the layout of scenic area facilities, and improve tourists' travel experience.

[0011] 2. Tourism management departments: Enable tourism management departments to grasp the operating trends of the tourism industry, achieve scientific, safe and intelligent management of the industry, and improve the level of public services.

[0012] 3. Tourist travel guidance: Provide tourists with real-time information on the number of tourists in scenic spots, help tourists plan their itineraries reasonably, avoid congestion, and improve the quality of travel.

[0013] 4. Tourism market analysis: Through the analysis and prediction of multimodal data, it provides data support for the research and analysis of the tourism market, helping relevant institutions to better understand market demand and trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of monitoring and prediction methods; Figure 2 It is a structural diagram of the deep neural network model in the present invention; Figure 3 It is the flow chart of the convolution process; Figure 4 This is a schematic diagram of timing processing. DETAILED DESCRIPTION

[0015] The present invention provides a passenger flow monitoring and prediction system based on multi-modal data such as mobile phone signaling data, traffic checkpoints, video surveillance, scenic spot wifi-hotspots, weather data, etc. through multi-model integration, which provides real-time passenger flow detection and effective prediction for scenic spots and tourism management departments, grasps the operating situation of the tourism industry, realizes scientific, safe and intelligent management of the industry by tourism management departments, and improves the public service capacity of tourists in tourist destinations.

[0016] The method for monitoring and predicting tourists in the park based on multimodal data analysis includes the following steps: S1, multimodal data extraction and preprocessing; use signaling data to obtain the number of users accessing the operator base stations around the scenic area, use equipment with pattern recognition algorithms to obtain information such as the vehicle models and numbers at the traffic checkpoints around the scenic area, connect to the surveillance video of the key areas of the scenic area, and use the video analysis algorithm to obtain information such as the number of tourists in the key areas of the scenic area. Obtain the scenic area wifi data through the interface to obtain the number of tourists accessing the scenic area WIFI system, extract the real-time weather information of the scenic area environmental detection equipment (environmental detection equipment that complies with the GB 3095-2018 standard) or authoritative departments (such as: China Weather Network Smart Service Platform http: / / www.weather.com.cn / wzfw / smart / index.shtml), and obtain the current environment and weather conditions of the scenic area.

[0017] S2, multimodal feature extraction and feature vector fusion Through the multi-modal data such as scenic spot signaling data, traffic checkpoints, video surveillance, scenic spot wifi hotspots, weather data, etc. provided by the operator, the maximum tourist carrying capacity of the scenic spot is used as the maximum value of the data and 0 is used as the minimum value. The noise, missing values ​​and outliers of the signaling data, traffic checkpoints, video surveillance, scenic spot wifi hotspot data, and weather data are preprocessed. Finally, the data is Z-score standardized. Z-score standardization is a common data processing method. The data is subtracted from its mean by attribute and divided by its standard deviation. The formula is: , The standardized value, is the original data, is the mean, is the standard deviation.

[0018] The data of each attribute is clustered around 0, and the variance is 1. The main purpose of this method is to eliminate the dimension effect of the data and make data of different magnitudes comparable. After the data is standardized, the eigenvalues ​​are used as their feature subvectors. Finally, they are concatenated in the order of {time, signaling data feature subvector, in-park vehicle feature subvector, wifi data feature subvector, video surveillance feature subvector, weather feature subvector} to form a complete feature vector.

[0019] S3, multi-model stacking integration The establishment of the model is divided into two parts: sub-model establishment and multi-model integration; the characteristics describing the real-time status of the scenic spot are input into the two sub-models, the decision tree model based on the gradient boosting algorithm and the deep neural network model, respectively, and an integrated model is established based on the prediction results of the two sub-models. Finally, the integrated model is used to calculate the real-time passenger flow of the scenic spot.

[0020] The model building method is as follows: (1) Establish a gradient boosting decision tree model Use LightGBM, a distributed gradient boosting framework based on the decision tree algorithm. (https: / / github.com / Microsoft / LightGBM) to build a gradient boosting decision tree model Input: Sample set , , where D is the sample collection time, which is in the format of yyyy-MM-ddHH:mm:ss, and n is the total number of samples. Observations are recorded every 10 minutes, there are 6 observations in an hour, and there are 144 (6x24) observations in a day. Given a fixed time, suppose you want to predict the temperature 1 hour in the future. To make this prediction, choose to use 5 days of observation time. Therefore, create a window containing the last 720 (5x144) observations to train the model.

[0021] Output: Gradient Boosted Decision Tree Model Initialize decision number ; Set the total number of decision trees trained by the gradient boosting decision tree model to M, and calculate the residual of each sample for m = 1, 2, ..., M respectively; then the residual of the sample is defined as ; , i=1, 2, ..., n; n is the total number of samples, and the residuals of each sample are fitted by using the regression decision tree model , learn the mth decision tree ,in is the parameter in the mth decision tree; Get the gradient boosting decision tree model , the output is the real-time number of tourists in the scenic area predicted by the gradient boosting decision tree model; (2) Establish a deep neural network model: Use the Keras framework to build a long short-term memory (LSTM) neural network model; standardize the signaling data, vehicle data in the park, WiFi data, video surveillance data, and weather data according to all the numerical data types in the feature data. When building the model, the batch dimension does not need to be displayed and is set to None to ensure that the input layer has the same shape as the training data set, that is, the shape of the input layer is [None, 14, 6], and then the dimension is adjusted to [None, 14, 6, 1] through reshape so that subsequent two-dimensional convolution operations can be performed. Set the number of parameter filters, the size of the convolution kernel, and regularization for the convolution layer, and set Padding=same to fill the input with 0 values. Using the BN layer to standardize the batch can speed up the training of the model, alleviate the vanishing gradient, and obey the normal distribution. After batch normalization, add a RELU excitation layer to make it easier to find the gradient and improve the convergence speed. Then use pooling to further amplify the feature information. At this time, the spatial relationship between the features is extracted after the convolution network is completed. After the convolution is completed, the convolution channel is adjusted to 1*1. After the convolution process is completed, the squeeze layer is used to reduce one dimension, so that the shape changes from four dimensions to three dimensions. Then, LSTM (where the first layer processes the input sequence and returns the output sequence of each time step, and the second layer processes the output sequence of the first layer and returns a single output of the entire sequence) is used to process the time series information of the data. Then a Dense layer containing 64 neurons is added to perform nonlinear transformation on the input data and learn the nonlinear relationship between the input features and the output. Finally, a fully connected layer is used as the output layer.

[0022] Model optimization: After the deep neural network model is built, the model is trained, and the Adam optimizer is used to make the gradient drop quickly to improve the training speed. At the same time, during the training process, the history will record the loss information and indicator information contained in each iteration, and judge whether the model is overfitting by visual display. In addition, during the training, the average value of the absolute error between the predicted value and the true value calculated by regression is used as the loss function. From the visual graph of loss, the loss and MSLE of the training set and the validation set are gradually decreasing, there is no overfitting phenomenon, and the network training is normal; the number of people in the park is calculated according to the characteristics of the test set, and then the Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Coefficient of determination R² (Rsquared, Coefficient of determination) evaluation indicators are used to judge the prediction of the model. The prediction indicators for time steps of T+1, T+3, and T+5 are saved as follows, indicating that the model fits the data well.

[0023] (3) Build a multi-model ensemble model based on the prediction results of the two sub-models: Use the scikit-learn library to create a decision tree regression model. For all samples in the training set , ,Will Input the trained gradient boosting decision tree model separately Get the prediction results And the deep neural network model g(T) to get the prediction results , construct a new training sample set for the integrated model , in the input space where the new sample set is located, the least squares method is used to segment the variables and Traverse and calculate the loss value of each segmentation point, select the segmentation variable and segmentation point with the smallest loss value, recursively divide each area into two sub-areas and determine the output value of each sub-area, and build a regression decision tree model. The output of the model is the predicted real-time number of tourists in the scenic area.

[0024] The data used by S1 include: signaling data, traffic checkpoint data, wifi-hotspot data, video surveillance data, and weather data. The details of the data are shown in the following table.

[0025] The S2 multimodal data analysis, feature extraction and feature fusion include: b1. Collect the number of tourists calculated based on signaling data provided by the scenic spot operator and extract the signaling data features; Where n represents the number of people accessing the nth base station. The data is normalized by Z-score to form the signaling data feature subvector ; b2. Analyze the traffic checkpoint data and extract the characteristics of vehicles in the park: extract the information of vehicles and models entering and leaving the scenic area through the traffic checkpoints, and remove the vehicles leaving from the data of vehicles entering; calculate the number of vehicles and models in the park, and then calculate the corresponding coefficients of models and number of people. Estimated number of people , Where n represents the model, represents the number of vehicle models entering the area. represents the number of the i-th vehicle type leaving the area. After Z-score normalization of the data, the characteristic sub-vector of vehicles in the park is formed. ; The corresponding relationship between vehicle type and number of people is as follows: b3. Analyze weather data and extract weather characteristics: divide weather into different levels according to weather conditions, and after Z-score standardization of the data, form a weather characteristic subvector ; As shown in the following table: b4. Analyze wifi hotspot data and extract wifi data features: obtain the number of online wifi terminals in all scenic spots and calculate the average of these data. After Z-score normalization of the data, the characteristic sub-vector of WiFi data is constructed. ; b4. Analyze video surveillance data and extract video surveillance data features: obtain real-time passenger flow data from passenger flow counting cameras in key areas of all scenic spots, and sum these data ,in Represents the number of tourists captured by the passenger flow camera. After the data is normalized by Z-score, it constitutes the feature subvector of the video surveillance data ; b5. Combine the signaling data feature sub-vector, the in-park vehicle feature sub-vector, the WiFi data feature sub-vector with the analyzed video surveillance data, and calculate the real-time number of in-park tourists in the scenic spot corresponding to the itinerary. The complete feature vector describing the number of tourists in the scenic spot in real time is composed as follows: ; A system for monitoring and predicting methods of tourists in the park based on multimodal data analysis, wherein the system is a deep neural network model and specifically includes: Part 1: Input layer, standardizes signaling data, in-park vehicle data, WiFi data, video surveillance data, and weather data according to all numerical types in the feature data to serve as input data for the model input layer.

[0026] Part 2: Reshape to adjust the dimension: The shape of the input layer is [None, 10, 6], and the dimension is adjusted to [None, 10, 6, 1] through reshape.

[0027] Part 3: BN+Relu convolution; the number of output channels of the convolution layer is 128, the size of the convolution kernel is 3x3 and L2 regularization, and Padding=same is set to fill the input with 0 values. Apply L2 regularization with a regularization coefficient of 0.01. Using the BN layer to normalize the batch can speed up the training of the model, alleviate the vanishing gradient and obey the normal distribution. Adding the RELU excitation layer after batch normalization makes it easier to find the gradient and improves the convergence speed. The processed data output is [None,14,6,128].

[0028] Part 4: Pooling sampling: The size of the pooling window is 2x2. Pooling further amplifies the feature information. The output data of the pooling process is [None, 14, 6, 128].

[0029] Part 5: Convolution Adjustment Channel Number: The output channel number of the convolution layer is 1, the size of the convolution kernel is 3x3 and L2 regularization, the regularization coefficient is 0.01, and the padding strategy Padding=same is set to fill the input with 0 values. The output data is [None,14,6,1].

[0030] Part 6: squeeze layer: Use the squeeze layer to remove the last dimension and adjust the dimension of the data from four dimensions to three dimensions. The output data is [None, 14, 4]; Part 7: The first LSTM layer: Use a bidirectional LSTM layer with 128 units, use ReLU as the activation function, and apply L2 regularization with a coefficient of 0.01; Part 8: The second LSTM layer: Use a bidirectional LSTM layer with 64 units, use ReLU as the activation function, apply L2 regularization, and the coefficient is 0.01; the output data is [None, 128].

[0031] Part 9: Fully connected layer: Use a fully connected layer with an output dimension of 64, use ReLU as the activation function, and the output data is [None,64].

[0032] Part 10: Output layer: A fully connected layer with an output dimension of 1 and an activation function of sigmoid is used as the output layer of the network. This data is the real-time number of tourists in the scenic area predicted by the deep neural network model.

[0033] The multi-model integration model is established based on the prediction results of the two sub-models. Data preparation: Load the dataset, for all samples in the training set , ,Will Input the trained gradient boosting decision tree model separately Get the prediction results And the deep neural network model g(T) to get the prediction results , construct a new training sample set for the integrated model .

[0034] Dataset division: Use the train_test_split function to split the dataset into a training set and a test set, where the test set accounts for 20% of the total dataset, and set the random seed random_state=42 to ensure the reproducibility of the results.

[0035] Model initialization: A regression decision tree model regressor is initialized using DecisionTreeRegressor. During the initialization process, some hyperparameters are specified, including the maximum depth of the decision tree max_depth=5, the minimum number of samples required for internal node splitting min_samples_split=20, the minimum number of samples required for leaf nodes min_samples_leaf=10, and the number of features used to find the best split max_features='sqrt'. The selection of these hyperparameters may need to be tuned through methods such as cross-validation to obtain the best model performance.

[0036] Model training: Use the training set (X_train, y_train) to train the model by calling the fit method. The model will learn how to predict the target variable y based on the training data.

[0037] Model prediction: Use the trained model to predict the test set X_test and get the prediction result y_pred.

[0038] Visualize the decision tree: Visualize the generated decision tree model by calling the plot_tree function. The visualized decision tree provides an intuitive understanding of the internal structure of the model, including the node division rules and the predicted values ​​of leaf nodes.

[0039] Model tuning: After using random search tuning, the optimal parameters are obtained: {'criterion': 'friedman_mse', 'max_depth': 16, 'max_features': 'sqrt', 'min_samples_leaf': 19, 'min_samples_split': 16}, mean square error (MSE): 3148.824295207477, root mean square error (RMSE): 56.114385813332014, mean absolute error (MAE): 45.52277965480213, coefficient of determination (R^2): 0.40567495144149135. The model performs well on the test set, and various evaluation indicators (such as MSE, RMSE, MAE and R^2) have reached a good level. This shows that the model can make relatively accurate predictions on the data.

[0040] Glossary Multimodal data: Multimodal data refers to a collection of information that contains multiple different data types or modalities. These modalities can include but are not limited to text, images, audio, video, sensor data, etc.

[0041] Signaling data: The signaling data in this article refers to the number of scenic spot tourists provided by the operator, which is calculated by filtering the selected scenic spot range according to the operator's rules.

[0042] Wi-Fi hotspot data: The Wi-Fi hotspot data in this article refers to the number of people accessing the Wi-Fi hotspot calculated after filtering by Wi-Fi\AP manufacturers according to established rules.

Claims

1. A method for monitoring and predicting tourists in the park based on multimodal data analysis, characterized by: The steps include: S1, multimodal data extraction and preprocessing; use signaling data to obtain the number of users accessing the operator base stations around the scenic area, use equipment with pattern recognition algorithms to obtain information such as the vehicle models and numbers at the traffic checkpoints around the scenic area, connect to the surveillance video of the key areas of the scenic area, obtain the number of tourists in the key areas of the scenic area through the video analysis algorithm, obtain the scenic area wifi data through the interface, obtain the number of tourists accessing the scenic area WIFI system, extract the real-time weather information of the scenic area environmental detection equipment that meets the GB 3095-2018 standard or the authoritative department, and obtain the current environment and weather conditions of the scenic area; S2, multimodal feature extraction and feature vector fusion Through the multi-modal data such as scenic spot signaling data, traffic checkpoints, video surveillance, scenic spot wifi hotspots, weather data, etc. provided by the operator, the maximum tourist carrying capacity of the scenic spot is used as the maximum value of the data and 0 is used as the minimum value. The noise, missing values ​​and outliers of the signaling data, traffic checkpoints, video surveillance, scenic spot wifi hotspot data, and weather data are preprocessed. Finally, the data is Z-score standardized. Z-score standardization is a common data processing method. The data is subtracted from its mean by attribute and divided by its standard deviation. The formula is: , where The standardized value, is the original data, is the mean, is the standard deviation; the data of each attribute is clustered around 0, and the variance is 1; the eigenvalues ​​after data standardization are used as their feature subvectors. Finally, they are concatenated in the order of {time, signaling data feature subvector, in-park vehicle feature subvector, wifi data feature subvector, video surveillance feature subvector, weather feature subvector} to form a complete feature vector; S3, multi-model stacking integration The establishment of the model is divided into two parts: sub-model establishment and multi-model integration; the characteristics describing the real-time status of the scenic spot are input into the decision tree model based on the gradient boosting algorithm and the deep neural network model respectively, and an integrated model is established based on the prediction results of the two sub-models. Finally, the integrated model is used to calculate the real-time passenger flow of the scenic spot; The model building method is as follows: (1) Establish a gradient boosting decision tree model Use LightGBM to build a gradient boosting decision tree model based on the distributed gradient boosting framework of the decision tree algorithm Input: Sample set , Where D is the sample collection time, n is the total number of samples; Output: Gradient Boosted Decision Tree Model Initialize decision number ; Set the total number of decision trees trained by the gradient boosting decision tree model to M, and calculate the residual of each sample for m=1, 2, ..., M respectively; then the residual of the sample is defined as; , , i=1, 2, ..., n; n is the total number of samples, and the residuals of each sample are fitted by using the regression decision tree model ; Learn and get the mth decision tree ,in is the parameter in the mth decision tree; Get the gradient boosting decision tree model , the output is the real-time number of tourists in the scenic area predicted by the gradient boosting decision tree model; (2) Establish a deep neural network model: Use the Keras framework to build a long short-term memory (LSTM) neural network model; standardize the signaling data, on-site vehicle data, WiFi data, video surveillance data, and weather data according to all the numerical data types in the feature data. When building the model, the batch dimension does not need to be displayed and is set to None to ensure that the input layer has the same shape as the training data set, that is, the shape of the input layer is [None, 14, 6], and then the dimension is adjusted to [None, 14, 6, 1] through reshape so that the subsequent two-dimensional convolution operation can be performed; set the number of parameter filters, the size of the convolution kernel, and regularization for the convolution layer, and set Padding=same to fill the input with zero values; Using the BN layer to standardize the batch can speed up the training of the model, alleviate the disappearance of the gradient and obey the normal distribution; adding the RELU excitation layer after batch normalization makes it easier to find the gradient and improves the convergence speed, and then using pooling to further amplify the feature information. At this time, after the convolution network is completed, the spatial relationship between the features is extracted; after the convolution is completed, the convolution channel is adjusted by 1*1; after the convolution process is completed, the squeeze layer is used to reduce one dimension, so that the shape changes from four dimensions to three dimensions, and then LSTM is used, in which the first layer processes the input sequence and returns the output sequence of each time step, and the second layer processes the output sequence of the first layer and returns a single output of the entire sequence to process the time series information of the data; Then add a Dense layer containing 64 neurons to perform nonlinear transformation on the input data and learn the nonlinear relationship between input features and output. Finally, use the fully connected layer as the output layer. A multi-model ensemble model is built based on the prediction results of the two sub-models: a decision tree regression model is created using the scikit-learn library; for all samples in the training set , ,Will Input the trained gradient boosting decision tree model separately Get the prediction results And the deep neural network model g(T) to get the prediction results , construct a new training sample set for the integrated model , in the input space where the new sample set is located, the least squares method is used to segment the variables and Traverse and calculate the loss value of each segmentation point, select the segmentation variable and segmentation point with the smallest loss value, recursively divide each area into two sub-areas and determine the output value of each sub-area, and build a regression decision tree model. The output of the model is the predicted real-time number of tourists in the scenic area.

2. The method for monitoring and predicting tourists in the park based on multimodal data analysis according to claim 1, characterized in that: The S2 multimodal data analysis, feature extraction and feature fusion include: b1. Collect the number of tourists calculated based on signaling data provided by the scenic spot operator and extract the signaling data. According to the characteristics; Where n represents the number of people accessing the nth base station. The data is normalized by Z-score to form the signaling data feature subvector b2. Analyze the traffic checkpoint data and extract the characteristics of vehicles in the park: extract the information of vehicles and models entering and leaving the scenic area through the traffic checkpoint respectively, and remove the vehicles leaving from the data of vehicles entering; calculate the vehicles and models in the park, and then calculate the corresponding coefficients according to the models and number of people. Estimated number of people , where n represents the model, represents the number of vehicle models entering the area. represents the number of the i-th vehicle type leaving the area. After Z-score normalization of the data, the characteristic sub-vector of vehicles in the park is formed. ; b3. Analyze weather data and extract weather characteristics: divide weather into different levels according to weather conditions, and after Z-score standardization of the data, form a weather characteristic subvector ; b4. Analyze wifi hotspot data and extract wifi data features: obtain the number of online wifi terminals in all scenic spots and calculate the average of these data. After Z-score normalization of the data, the characteristic sub-vector of WiFi data is formed ; b5. Analyze video surveillance data and extract video surveillance data features: obtain real-time passenger flow data from passenger flow counting cameras in key areas of all scenic spots, and sum these data ,in Represents the number of tourists captured by the passenger flow camera. After the data is normalized by Z-score, it constitutes the feature subvector of the video surveillance data ; b6. The signaling data feature vector, the vehicle feature vector, the WiFi data feature vector and the analysis of video surveillance data splicing, and the corresponding scenic spot with the trip in real time in the park number of tourists The complete feature vector describing the number of tourists in the scenic spot in real time is composed as follows: .

3. The system of the method for monitoring and predicting tourists in the park based on multimodal data analysis according to claim 1 or 2, characterized in that: The system is a deep neural network model; Part 1: Input layer: standardize the signaling data, in-park vehicle data, WiFi data, video surveillance data, and weather data according to all the numerical data types in the feature data to serve as the input data for the model input layer; Part 2: Reshape to adjust the dimension: The shape of the input layer is [None, 10, 6], and the dimension is adjusted to [None, 10, 6, 1] through reshape; Part 3: BN+Relu convolution; the number of output channels of the convolution layer is 128, the size of the convolution kernel is 3x3 and L2 regularization, and Padding=same is set to fill the input with 0 values; Apply L2 regularization with a regularization coefficient of 0.

01. Use the BN layer to normalize the batches to speed up the model training, alleviate the vanishing gradient, and make it obey the normal distribution. Add a RELU excitation layer after batch normalization, and the processed data output is [None, 14, 6, 128]; Part 4: Pooling sampling, the size of the pooling window is 2x2, pooling further amplifies the feature information, and the output data of the pooling process is [None, 14, 6, 128]; Part 5: Convolution adjusts the number of channels. The number of output channels of the convolution layer is 1, the size of the convolution kernel is 3x3 and L2 regularization is used, the regularization coefficient is 0.01, and the padding strategy Padding=same is set to fill the input with 0 values. The output data is [None, 14, 6, 1]; Part 6: squeeze layer, use the squeeze layer to remove the last dimension, adjust the dimension of the data from four dimensions to three dimensions, and the output data is [None, 14, 4]; Part 7: The first LSTM layer uses a bidirectional LSTM layer with 128 units, uses ReLU as the activation function, and applies L2 regularization with a coefficient of 0.01; Part 8: The second LSTM layer uses a bidirectional LSTM layer with 64 units, uses ReLU as the activation function, applies L2 regularization, and the coefficient is 0.01; the output data is [None, 128]; Part 9: Fully connected layer, using a fully connected layer with an output dimension of 64, using ReLU as the activation function, and the output data is [None, 64]; Part 10: Output layer, a fully connected layer with an output dimension of 1 and an activation function of sigmoid. As the output layer of the network, the real-time number of tourists in the scenic area is predicted using a deep neural network model.

Images