A multi-modal crowd density prediction method based on time convolution network

Abstract

The application discloses a kind of multi-modal crowd density prediction methods based on time convolution network, comprising the following steps: video information module obtains image from monitoring camera, the image obtained is carried out crowd counting by crowd counting model, obtains the crowd density of each camera every time, and is organized into time series data;The time series data of crowd density is respectively extracted into the current time hidden vector for each sub-region by time convolution network;Itinerary planning module extracts corresponding itinerary information from itinerary planning table for multi-modal prediction;The features of fusion module and video information module are fused, and the crowd density prediction value of each sub-region in the future is obtained.This application can improve the prediction accuracy of the model by fusing multi-modal information, and can give a smooth prediction result when facing low SNR data, and can also respond in time when the crowd density changes suddenly.

Description

Technical Field

[0001] This invention relates to two major fields: deep learning and crowd density prediction, and mainly to a bimodal crowd density prediction model for large transportation venues. Background Technology

[0002] As the economy develops and people's travel needs increase, large shopping malls, commercial centers, airports, high-speed rail stations, bus stations, sports centers, concert venues, and even some tourist attractions are experiencing ever-increasing visitor volumes. This has led to safety concerns that have drawn public attention. For example, the Shanghai Bund stampede occurred precisely because of excessive crowd density; the monitoring methods at the time failed to detect the abnormal surge in crowds in time, resulting in a series of tragedies. To mitigate these risks and protect people's lives, academic researchers have successively proposed crowd modeling techniques for various scenarios.

[0003] Crowd modeling is of great significance in today's society. With continuous economic development, people's demand for travel and tourism is increasing, leading to a surge in visitor traffic at large shopping malls, commercial centers, airports, high-speed rail stations, bus stations, sports centers, concert venues, and even some tourist attractions. This has raised concerns about potential safety hazards. Currently, methods related to crowd modeling mainly focus on crowd simulation, crowd trajectory prediction, and traffic forecasting. These methods either fail to fully utilize real-time data, focus only on short-term, single-camera predictions, or only address city-scale predictions. Data sets and models for long-term, multi-camera crowd density prediction in large indoor scenes based on video data have yet to be developed.

[0004] Among these, crowd monitoring and control in large indoor settings is an urgent need for the industry. In recent years, the number of large buildings such as transportation facilities and shopping malls in China has increased year by year, leading to greater pressure on crowd control. For example, as of August 2019, the terminal area of ​​Guangzhou Baiyun International Airport reached 1.18 million square meters, and its passenger throughput in 2020 reached 43.76 million. The airport terminal has hundreds of surveillance cameras. Monitoring such a large number of cameras manually would consume a significant amount of manpower. Furthermore, since the human eye cannot simultaneously monitor multiple cameras, oversights are easily made, making it impossible to anticipate and prevent potential safety hazards in advance.

[0005] Based on the aforementioned needs, this invention focuses on researching crowd density prediction in large indoor scenes. By using a deep learning model to aggregate information from all surveillance cameras and multimodal data within the scene, it aims to predict future crowd density in various zones of the indoor scene, thereby supporting precise crowd control.

[0006] Existing technologies (LIU W, YANG Y, ZHONG J. Towards Dual-Modal Crowd Density Forecasting in Transportation Building; procedures of the 2022 International Joint Conference on Neural Networks (IJCNN), F, 2022 [C]. IEEE.) suffer from insufficient prediction accuracy and poor generalization performance when dealing with crowd density data with high signal-to-noise ratio. Summary of the Invention

[0007] This invention addresses the problem of crowd density prediction based on video data, proposing a multimodal crowd density prediction method for large transportation venues. It applies a spatiotemporal convolutional network (TCN) to the bimodal crowd density prediction problem in indoor scenes, utilizing both surveillance video information and travel plan information. The method uses a temporal convolutional network (TCN) as the temporal correlation extraction module for each part, predicting the crowd density trend at least 30 minutes in advance using historical crowd data obtained from multiple cameras.

[0008] The present invention is achieved by at least one of the following technical solutions.

[0009] A multimodal crowd density prediction method based on temporal convolutional networks includes the following steps:

[0010] The video information module acquires images from the camera, and the acquired images are processed by a crowd counting model to count the crowd, obtain the crowd density of each camera at each time and each sub-region, and organize it into time series data; the time series data is processed by a temporal convolutional network to extract the latent vector of the current time.

[0011] The itinerary planning module extracts relevant itinerary information from the itinerary planning table for multimodal prediction;

[0012] The fusion predictor module integrates the features from the trip planning module and the video information module, and predicts the population density of each sub-region at multiple time steps based on the fused features.

[0013] Furthermore, the crowd counting model counts the crowd density in an image by including:

[0014] The area captured by each camera is defined as a sub-region. For sub-region v, the crowd density at time t is... Then the crowd density vector of all cameras at time t is: Where V represents the total number of sub-regions in the scene. The population density at time t refers to the population density of the (V-1)th sub-region, where v = V-1. This population density vector serves as the input feature of the temporal convolutional network.

[0015] Furthermore, the temporal convolutional network extracts the latent vector at the current time step, including the following steps:

[0016] For time t, obtain a value of size T. his The population density vector of the historical time window is denoted as the input matrix. in For tT his The crowd density vector at any given time;

[0017] The input matrix is ​​passed through L temporal convolutional layers to extract high-dimensional features; the kernel size of the temporal convolutional layers is denoted as K. t The number of convolution kernels is C hidden The relationship between the convolutional layers is as follows:

[0018]

[0019] in Let l be the latent vector of the l-th convolutional layer at time t. When l = 0, The input matrix; Temporal convolution kernel; * d This is a dilated convolution operation with a dilation rate of d; σ(·) is the activation function, and the previous layer's hidden vector... Connect to the next layer via residuals; b is the bias term. This indicates that the elements in Γ are real numbers and the size is C. hidden ×K t ×1;

[0020] After convolution, the features are compressed to a temporal dimension of 1 through temporal convolution in the output layer. At this point, each sub-region v has only one latent vector. It represents the historical population density trend, and is finally obtained as the output matrix after multiple convolution operations. in The latent vector representing the historical population density trend of the (V-1)th sub-region express The elements in the array are real numbers, and its size is C. hidden ×1×V;

[0021] Finally, after an upsampling layer, the latent vector with a time dimension of 1 is expanded to the number of time steps T to be predicted. pred It serves as the input for the video information module.

[0022] Furthermore, the itinerary planning module extracts relevant itinerary information from the itinerary planning table for multimodal prediction, including the following steps:

[0023] S1. First, through data preprocessing, each flight in the itinerary schedule table is converted into a fixed-size itinerary schedule signal;

[0024] S2. The trip plan signal extracts high-level features through multiple temporal convolutional layers, distributes the trip information to the sub-regions corresponding to each camera, and associates the trip signal with the crowd density of each sub-region.

[0025] Furthermore, a Gaussian kernel function is used to convert each trip information in the trip schedule table into a trip schedule signal:

[0026]

[0027] Where α s Let μ be the number of passengers on a given trip; μ is the mean of the Gaussian kernel function; σ controls the variance of the Gaussian distribution; p s (t) represents the trip plan signal at time t, and the trip plan signals are summed according to the checkpoint set to which each trip belongs.

[0028] Further, step S2 includes:

[0029] For the predicted time step p, with the future T s The time window for capturing the trip plan signal at each time step corresponds to the 3D signal tensor as follows: in This represents the trip plan signal matrix for all checkpoint sets at time t;

[0030] C sched V represents the number of signal channels; check Indicates the number of trips;

[0031] Different prediction time steps share the same set of network weights, and after temporal convolution, the output is the travel information features of each checkpoint set at time step p. That is, the size is C hidden ×1×V check Real tensors;

[0032] Travel information characteristics of checkpoint sets Through pooling operations on the weight allocation matrix, the input travel information features are linearly combined with multiple sets of different weights, ultimately forming multiple output latent vectors. Each output latent vector corresponds to a sub-region in the scene, and the mapping between the checkpoint set and the sub-region is considered as an allocation matrix. For the element α in the m-th row and n-th column of the allocation matrix m , nThis indicates that passengers at the m-th checkpoint set will have α. m , n The probability of reaching the nth sub-region.

[0033] Furthermore, the pooling operation using the weight allocation matrix is ​​as follows:

[0034]

[0035] in, This represents the trip plan feature at time step p after weighted pooling, which is a feature of size C. hidden A real tensor of ×1×V, where V represents the total number of sub-regions in the scene, and C... hidden The number of convolution kernels is represented; σ(·) is the activation function, and W is the parameter matrix of the linear transformation. The matrix obtained by softmax calculation using parameter matrix A is denoted as a. m , n Let be the element in the m-th row and n-th column of matrix A.

[0036] Furthermore, element α m , n The calculation is performed using the following softmax formula:

[0037]

[0038] Where V represents the total number of sub-regions in the scene, the original matrix A consists of parameters learned during the training phase, and a m , j represents the element in the m-th row and j-th column of matrix A.

[0039] Furthermore, the fusion predictor module employs a splicing-based fusion method to fuse information:

[0040]

[0041] In the formula This represents the fusion vector at prediction time step p and sub-region v; is the video information vector for sub-region v; C is the travel plan vector for predicting time step p and sub-region v; hidden Indicates the number of convolution kernels; This indicates a size of 2C. hidden A real column vector;

[0042] The input features of the predictor in the fusion predictor module are passed through three temporal convolutional layers, and the output intermediate latent features are equal in size in the temporal dimension. The zero-padding of each layer is expressed as follows:

[0043] zero padding size = (K t -1)×d

[0044] In the formula K t d represents the kernel size of the temporal convolutional layer, and d is the dilation rate.

[0045] Finally, the output vector of the temporal convolutional layer is passed through a multilayer perceptron (MLP) to output a set of predicted crowd density values. in T represents the predicted population density value for subregion v at prediction time step p. pred Indicates the number of time steps to perform the prediction; The dimension is T. pred A real column vector.

[0046] Furthermore, for sub-region v, the input features of the predictor in the fusion predictor module are:

[0047]

[0048] in Let T represent the T-th fusion vector, where T = T pred C hidden This indicates the number of convolution kernels.

[0049] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0050] This invention addresses the sensitivity of crowd density to external conditions in indoor scenes by fusing multimodal information from indoor environments. It enables the prediction of future events and proposes using pooling techniques with an assignment matrix to learn the correlation between video and travel plan information. The fusion of multimodal information in this invention improves the model's prediction accuracy, provides stable predictions even with low signal-to-noise ratio data, and responds promptly to sudden changes in crowd density.

[0051] This invention modifies the network structure based on existing technologies, employing a temporal convolutional network entirely for extracting temporal correlations. Furthermore, it utilizes a bottleneck structure of compression-expansion—compressing the time dimension to 1 before upsampling—to improve the generalization performance of the model and enhance its prediction accuracy. This invention achieves more stable prediction results in high signal-to-noise ratio crowd density data. Attached Figure Description

[0052] Figure 1 This is an overall structural diagram of a multimodal crowd density prediction method based on temporal convolutional networks, as shown in the embodiment.

[0053] Figure 2This is a network structure diagram of the video information module in an embodiment;

[0054] Figure 3 This is a schematic diagram of a travel planning signal for an example.

[0055] Figure 4 Here is a network structure diagram of the itinerary planning module for an example;

[0056] Figure 5 The example uses a pooling layer based on a weight allocation matrix;

[0057] Figure 6 Here is a network structure diagram of the fusion predictor module in an embodiment;

[0058] Figure 7 This is a schematic diagram of the temporal convolutional layer structure of the fusion predictor in an embodiment. Detailed Implementation

[0059] The method of the present invention will now be further described with reference to the accompanying drawings.

[0060] A multimodal crowd density prediction method based on temporal convolutional networks includes the following steps: a video information module acquires images from surveillance cameras, the acquired images are processed by a crowd counting model to count the crowd density at each moment of each camera, and the data is organized into time series data; the time series data of crowd density is processed by a temporal convolutional network to extract the latent vector of the current moment for each sub-region.

[0061] The itinerary planning module extracts relevant itinerary information from the itinerary planning table for multimodal prediction;

[0062] The itinerary planning module extracts relevant itinerary information from the itinerary planning table for multimodal prediction;

[0063] The fusion predictor module integrates the features from the trip planning module and the video information module, and predicts the population density of each sub-region at multiple time steps based on the fused features.

[0064] This embodiment of a multimodal crowd density prediction method based on temporal convolutional networks is divided into four sub-modules. The model part is divided into three sub-modules: video information module, travel information module, and fusion predictor module, plus a training and testing sub-module for training the network.

[0065] As a preferred embodiment, such as Figure 1As shown, the video information module first acquires images from the surveillance cameras at 5-minute intervals. The acquired images are then processed by a crowd counting model to count the crowd density at each moment for each camera, and this data is organized into time-series data. The time-series data of the crowd density is then processed by a temporal convolution module to extract the latent vector for each sub-region at the current moment. This module contains the historical trend of crowd density extracted from the video data.

[0066] The itinerary planning module is a new module used to extract relevant itinerary information from itinerary schedules (such as train schedules, flight schedules, etc.) for multimodal prediction. First, it preprocesses each flight in the itinerary schedule table into a fixed-dimensional itinerary plan signal. Then, the itinerary plan signal is passed through a temporal convolutional layer to generate a latent vector for the corresponding time moment, serving as the feature of the itinerary plan. Subsequently, a pooling layer based on a weighted allocation matrix learns the correlation between the latent vectors from the video information module and the itinerary planning module. This module pools each itinerary plan signal using a weighted matrix, distributing the itinerary plan information to each corresponding sub-region through a weighted summation using different weight coefficients. These weight coefficients are learned from the training data.

[0067] Finally, the fusion predictor module fuses the features of the fusion module and the video information module, and through multi-step prediction, obtains the predicted values ​​of the crowd density at multiple future times for each sub-region.

[0068] The training and testing module is used to train the parameters of the above network modules and to test the predictive performance of the model.

[0069] The video information module's function is to extract crowd density information from video data. First, it acquires one frame of image information from the video data stream of the surveillance cameras at fixed intervals. The acquired image information is then used to count the crowd density in the image using a crowd counting model (such as CSRNet). In a large indoor building, there are multiple cameras, and the area captured by each camera is defined as a sub-region. For sub-region v, the crowd density value at time t is... Therefore, the crowd density vector of all cameras in the entire indoor building at time t is: This vector serves as the input feature of the neural network described below, where V represents the total number of sub-regions in the scene.

[0070] like Figure 2 As shown, the deep learning model structure used in the video information module is illustrated. The video information module employs a temporal convolutional network to extract the temporal correlation of the input features. First, for time t, the neural network obtains T... his The population density vector of the historical time window of size is denoted as the input matrix. Then, the input matrix is ​​passed through L temporal convolutional layers to extract high-dimensional features. The kernel size of the temporal convolutional layers is denoted as K. t The number of convolution kernels is denoted as C. hidden The relationship between the convolutional layers can be expressed by formula (1):

[0071]

[0072] in Let be the latent vector of the l-th convolutional layer at time t. In particular, when l = 0, That is, the input matrix X t ; Temporal convolution kernel; * d This is a dilated convolution operation with a dilation rate of d; σ(·) is the activation function. The hidden state of the previous layer... Connect to the next layer through residuals.

[0073] Then, after convolution, the features are compressed to a temporal dimension of 1 through a temporal convolution operation in the output layer. At this point, each sub-region v has only one latent vector. This represents the historical population density trend, specifically the output matrix obtained after multiple convolutional operations.

[0074] Finally, after the upsampling layer, the original latent vector with a time dimension of 1 is expanded to the number of time steps T to be predicted. pred This is used as the output of the video information module. The resulting output matrix is ​​then obtained. Following the feature extraction, this invention adds an upsampling layer to prepare for the subsequent temporal convolution-based fusion predictor. This allows the predictor to access all historical information hidden in the latent vectors at each prediction time step. Furthermore, the time dimension is compressed and then expanded, effectively creating a bottleneck structure that filters out time-related information from the data, resulting in better generalization performance.

[0075] Overall, all modules of the method in this invention are entirely based on temporal convolutional networks. Furthermore, the video information module first compresses the temporal dimension of the features to 1, and then expands it using an upsampling layer. This bottleneck structure of compression-expansion in the temporal dimension allows the model to achieve more stable prediction results in high signal-to-noise ratio (SNR) crowd density data. It significantly improves generalization performance compared to existing technologies, specifically in terms of improved prediction accuracy and better generalization performance when dealing with high SNR crowd density data.

[0076] The itinerary planning module is used to extract relevant itinerary information from itinerary planning tables (such as train schedules, flight schedules, etc.) for multimodal prediction. It first converts each flight in the itinerary planning table into a fixed-size itinerary planning signal through data preprocessing. Specifically, taking airport flight information as an example, flight information consists of various information such as flight number, check-in gate, departure time, and number of passengers. An example is shown in Table 1 below.

[0077] Table 1. Sample Flight Information

[0078]

[0079] In order to utilize the information in the itinerary schedule, feature engineering is required on the flight schedule. First, the flight information is converted into corresponding time-series signals, hereinafter referred to as itinerary schedule signals. A corresponding signal is generated for each itinerary using a Gaussian kernel function, as shown in formula (2):

[0080]

[0081] Where α s σ represents the number of passengers on a given trip; μ is the mean of the Gaussian kernel function; σ controls the variance of the Gaussian distribution, representing the duration for which a given trip affects the state of the indoor crowd.

[0082] Here, we need to introduce the concept of a checkpoint set. A checkpoint is a location where passengers need to arrive and undergo corresponding inspection procedures, such as check-in gates and security checkpoints in an airport terminal. Because each itinerary may have multiple checkpoints, these multiple checkpoints are called a checkpoint set. For example, the checkpoint set for AC Airlines flights in Table 1 is S. AC = {E, F}.

[0083] All travel planning signals generated by each travel are summed according to their respective checkpoint sets. Since the checkpoint set is typically finite, this method allows the varying number of travels to form a fixed-size characteristic. To visually represent this idea, the generated signal characteristics can be represented by an appendix. Figure 3 This is represented in the graph. Each signal curve in the graph corresponds to the travel plan signal of its respective checkpoint set. The horizontal axis of the signal represents time, and the vertical axis (amplitude) indicates the number of passengers at each moment. Each signal curve contains multiple peaks, representing a mixture of multiple travel signals. Figure 3 μ represents the time of the peak, which is a value calculated based on the flight's departure time.

[0084] It's important to note that this approach isn't limited to flight information. Since high-speed rail and train travel information operate on similar models, this method can be extended to other similar transportation infrastructure scenarios.

[0085] Subsequently, the itinerary planning information is processed through multiple temporal convolutional layers to extract high-level features, employing a sliding window mechanism, such as... Figure 4 As shown. Specifically, for each prediction time step, the model uses the future T... s The time window for capturing the trip plan signal at each time step corresponds to the 3D signal tensor as follows: in This represents the trip plan signal matrix for all checkpoint sets at time t; Used to represent S (p) For size C sched ×T s ×V check The real tensor. Used to indicate For a size of V check ×C sched A real matrix; C sched This represents the number of channels in the signal. Here, you can choose to calculate an additional first-order difference value for the signal, adding it to the original signal's channels, resulting in C. sched =2.

[0086] Secondly, as the prediction time increases, the signal window slides forward and is input into the network again to obtain the travel information for the next time window. Different prediction time steps p

[0087] Sharing the same set of network weights, and because the network uses a one-dimensional temporal convolution kernel, different sub-regions also essentially share the same set of convolution kernel parameters. Finally, the convolution outputs p, representing the travel information features of each checkpoint set. Different time steps p share a set of network weights.

[0088] To distribute travel information to the sub-regions corresponding to each camera, it's necessary to find the correlation between these travel signals and the crowd density of each sub-region. Here, a pooling mechanism based on a weight allocation matrix is ​​used, such as... Figure 5 As shown, Figure 5 The squares on the lower left of the image correspond to the input latent vectors of different checkpoint sets. Through pooling operations on the weight allocation matrix, the input latent vectors are linearly combined with multiple sets of different weights, ultimately forming multiple output latent vectors, each corresponding to a sub-region in the scene. The mapping between the checkpoint set and the sub-regions is considered as an allocation matrix. For the element α in the m-th row and n-th column of the matrix m，n This indicates that passengers at the m-th checkpoint set will have α. m，n The probability of reaching the nth sub-region is [not specified]. Then, pooling operations based on the weight allocation matrix can be expressed by the following formula:

[0089]

[0090] Where σ(·) is the activation function and W is the parameter matrix of the linear transformation. The matrix is ​​obtained by performing softmax calculation on another parameter matrix A, let a. m，n Let be the element in the m-th row and n-th column of matrix A. express It is a real matrix of size V check ×V. element α m，n The calculation is performed using the following softmax formula:

[0091]

[0092] Specifically, the original matrix A represents the parameters that can be learned during the training phase, a m，j Let represent the element in the m-th row and j-th column of matrix A, where j is a variable of the summation symbol ∑.

[0093] Pooling based on the weight allocation matrix means that the model can determine the degree of attention each sub-region pays to different travel plan information features based on the coefficients of the weight matrix. The model can learn the corresponding weight values ​​from all training data, thereby learning the potential correlation patterns between the population density distribution of sub-regions and the travel information of checkpoint sets.

[0094] The fusion predictor is used to predict the crowd density in different areas of an indoor scene. To perform multimodal prediction, features from the video information module and the travel plan information module need to be fused. A stitching-based fusion method is used to fuse the bimodal information, as shown in formula (5).

[0095]

[0096] In the formula This represents the fusion vector of sub-region v at prediction time step p; is the video information vector for sub-region v; This is the travel plan vector for predicting time step p and sub-region v. express It is a size 2C hidden A real column vector.

[0097] The subsequent network structure of the fusion predictor is as follows: Figure 6 As shown. For sub-region v, the input features of the predictor are T pred The above-mentioned fused vectors are composed of, denoted as... T pred This corresponds to the number of time steps in the predictor. First, it passes through three temporal convolutional layers, each employing causal convolution, and each layer uses a different kernel size K. t One-dimensional dilated convolution kernels with different dilation rates d, and the number of convolution kernels in each layer is C. hidden Furthermore, the features of each layer are left-zero-padded in the time dimension to ensure that the time dimension of the output features of each temporal convolutional layer remains unchanged, such as... Figure 7 As shown, the input features of the predictor, after passing through three convolutional layers, produce intermediate latent features of equal size in the time dimension, which is T. pred The formula for zero-fill in each layer can be expressed as follows:

[0098] zero padding size = (K t -1)×d

[0099] Finally, the output vector of the convolutional layer is passed through a multilayer perceptron (MLP) to output a set of predicted values ​​for the final crowd density: in This represents the predicted population density value for sub-region v at prediction time step p. Different sub-regions v use shared network weights for calculation. The dimension is T. pred A real column vector.

[0100] As one example, the data source for the training and testing module is Terminal 1 of Guangzhou Baiyun International Airport. The video data collection information is as follows:

[0101]

[0102]

[0103] It is worth noting that, in order to support predictions of crowd density over longer periods (over 30 minutes) to meet actual business needs, the data collection interval τ was designed to be 5 minutes. This allows the model to cover prediction durations of over 30 minutes through predictions at more than 6 time steps.

[0104] The itinerary information, which is the same as flight information, is collected from the corresponding time period within the same scenario.

[0105] 80% of the collected data is used as the training set, 10% as the validation set, and the remaining 10% as the test set. The network structure and training hyperparameters are shown in the table below:

[0106]

[0107] During training, the loss function is calculated using the MAE formula:

[0108]

[0109] The Adam optimizer was used as the optimizer during model training. The training iterations were performed 50 times, and the model with the lowest loss value on the validation set was selected as the test model. N represents the number of sample points, Y... i This represents the true value of population density. This represents the predicted population density value of the network. The sample points do not distinguish between time t and sub-region v, and each population density value is a sample point.

[0110] The final results show that both training and validation errors were effectively reduced to low values ​​during the training process, indicating that the trained model has good accuracy and generalization ability. Test results show that, due to the inclusion of travel planning information, the model achieves higher prediction accuracy than when based solely on video information. Furthermore, case studies demonstrate that the model not only provides stable prediction results even with low signal-to-noise ratio data, but also effectively anticipates and responds to sudden changes in crowd density caused by external conditions. This demonstrates that the modeling for crowd density prediction in indoor scenes presented in this invention is highly effective and can adapt to the challenges of indoor scenes being sensitive to external conditions and having low signal-to-noise ratio crowd density data.

[0111] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multimodal crowd density prediction method based on temporal convolutional networks, characterized in that, Includes the following steps: The video information module acquires images from the camera. The acquired images are processed by a crowd counting model to count the crowd density at each time step of each sub-region of each camera, and then organized into time series data. The time series data is processed by a temporal convolutional network to extract the latent vector at the current time step. After convolution, the features are processed by a temporal convolution operation in the output layer to compress the temporal dimension of the features to 1. At this point, each sub-region is represented by only one latent vector, which represents the historical crowd density trend. Finally, after multiple convolution operations, the output matrix is ​​obtained. Finally, after an upsampling layer, the latent vector with a time dimension of 1 is expanded to the time step to be predicted, which is then used as the input to the video information module. The itinerary planning module extracts relevant itinerary information from the itinerary planning table for multimodal prediction; The fusion predictor module fuses the features of the trip planning module and the video information module, and predicts the population density of each sub-region at multiple time steps in the future based on the fused features. The input features of the predictor in the fusion predictor module are passed through three temporal convolutional layers, and the output intermediate latent features are equal in size in the temporal dimension. The zero-padding of each layer is expressed as follows: In the formula The kernel size of the temporal convolutional layer. The expansion rate; For sub-regions The input features of the predictor in the fusion predictor module are: ， in Indicates the first A fusion vector, = , Indicates the number of convolution kernels. Indicates the size of the vector.

2. The multimodal crowd density prediction method based on temporal convolutional networks according to claim 1, characterized in that, Crowd counting models count the density of people in an image, including: The area captured by each camera is defined as a sub-region. ,exist The crowd density value at time t is Then all cameras are The crowd density vector at time t is ,in This represents the total number of sub-regions in the scene. Refers to the first Sub-regions Crowd density at any given time The crowd density vector serves as the input feature of the temporal convolutional network.

3. The multimodal crowd density prediction method based on temporal convolutional networks according to claim 1, characterized in that, Temporal convolutional networks extract the latent vector at the current time step, including the following steps: For time Get the size as The population density vector of the historical time window is denoted as the input matrix. ,in for The crowd density vector at any given time; The input matrix is ​​processed Temporal convolutional layers extract high-dimensional features; the kernel size of the temporal convolutional layer is denoted as . The number of convolution kernels is The relationship between the convolutional layers is as follows: in For the first Layer convolution at time The latent vector of time, when hour, The input matrix; Temporal convolution kernel; The parameter is the expansion rate. The dilated convolution operation; For the activation function, the hidden vector of the previous layer Connect to the next layer via residuals; For bias terms, express The elements in the array are real numbers, and the size is [size missing]. .

4. The multimodal crowd density prediction method based on temporal convolutional networks according to claim 1, characterized in that, The itinerary planning module extracts relevant itinerary information from the itinerary planning table for multimodal prediction, including the following steps: S1. First, through data preprocessing, each flight in the itinerary schedule table is converted into a fixed-size itinerary schedule signal; S2. The trip plan signal extracts high-level features through multiple temporal convolutional layers, distributes the trip information to the sub-regions corresponding to each camera, and associates the trip signal with the crowd density of each sub-region.

5. The multimodal crowd density prediction method based on temporal convolutional networks according to claim 4, characterized in that, The Gaussian kernel function is used to convert each trip information in the trip plan table into a trip plan signal: in The number of passengers on a given trip; It is the mean of the Gaussian kernel function; Controlling the variance of the Gaussian distribution, Represents time The time-based travel plan signals are summed based on the checkpoint set to which each travel belongs.

6. The multimodal crowd density prediction method based on temporal convolutional networks according to claim 2, characterized in that, Step S2 includes: For predicting time steps In the future The time window for capturing the trip plan signal at each time step corresponds to the 3D signal tensor as follows: ,in This indicates that the set of all checkpoints is at time [time]. The itinerary planning signal matrix; Indicates the number of signal channels; Indicates the number of trips; Different prediction time steps share the same set of network weights, which are then output after temporal convolution. Travel information characteristics of each checkpoint set in the time step , That is, the size is Real tensors; Travel information characteristics of checkpoint sets Through pooling operations on the weight allocation matrix, the input travel information features are linearly combined with multiple sets of different weights, ultimately forming multiple output latent vectors. Each output latent vector corresponds to a sub-region in the scene, and the mapping between the checkpoint set and the sub-region is considered as an allocation matrix. For the first element in the allocation matrix m row and number n Column elements Indicates the first m Passengers at each checkpoint set will have The probability of reaching the first n Sub-regions.

7. The multimodal crowd density prediction method based on temporal convolutional networks according to claim 6, characterized in that, Pooling operation using the weight allocation matrix is ​​as follows: in, Indicates at time step The trip plan feature after weighted pooling is a feature of size 1. real tensors, This represents the total number of sub-regions in the scene. Indicates the number of convolution kernels; It is an activation function. The parameter matrix for the linear transformation, Through parameter matrix The matrix obtained after softmax calculation is denoted as... For matrix The Middle Line number The elements of the column.

8. The multimodal crowd density prediction method based on temporal convolutional networks according to claim 7, characterized in that, elements in The calculation is performed using the following softmax formula: in Represents the total number of sub-regions in the scene; the original matrix. These are the parameters learned during the training phase. Representation matrix The Middle m Line number j The elements of the column.

9. The multimodal crowd density prediction method based on temporal convolutional networks according to any one of claims 1 to 8, characterized in that, The fusion predictor module uses a splicing-based fusion method to fuse information: In the formula Indicates the time step in the prediction Subregion The fusion vector; sub-region The video information vector; For predicting time steps Subregion The itinerary planning vector; Indicates the number of convolution kernels; Indicates a size of 2 A real column vector; Finally, the output vector of the temporal convolutional layer is passed through a multilayer perceptron to output a set of predicted values ​​for the final crowd density. ,in Indicates the prediction time step is sub-region The predicted population density value, Indicates the number of time steps to perform the prediction; Indicates the size is A real column vector.

Images