A subway passenger flow prediction and management method, a storage medium and an electronic device

Abstract

This invention relates to a method, storage medium, and electronic device for predicting and controlling subway passenger flow, belonging to the field of urban rail transit operation and control technology. The method includes: generating a multimodal fusion feature vector; mapping each pedestrian in the panoramic video of the entire station to a Gaussian distribution using pedestrian kernel density estimation and superimposing all Gaussian distributions to obtain a superimposed result; correcting the superimposed result through occlusion adaptive weights and performing density adaptive Gaussian smooth interpolation; inputting the multimodal fusion feature vector of historical time series into a pre-constructed time series prediction model to obtain the predicted total passenger flow and the predicted regional passenger flow density distribution for future times; and jointly determining the station passenger flow control level based on the pixel-level passenger flow density heatmap, the predicted total passenger flow, and the predicted regional passenger flow density distribution, and generating a corresponding control plan, thereby improving the accuracy of passenger flow prediction and the reliability of control decisions.

Description

Technical Field

[0001] This invention relates to the field of urban rail transit operation and management technology, and in particular to a method for predicting and managing subway passenger flow, a storage medium, and an electronic device. Background Technology

[0002] Large passenger flow management in subway stations relies on the fusion, perception, and prediction of multi-source data. Existing technologies typically collect multimodal data such as passenger flow videos, turnstile statistics, and external correlations, and use a fixed weighting method to fuse various types of data to support subsequent passenger flow prediction and management decisions.

[0003] However, in the actual operation environment of subway stations, the data quality of different modalities fluctuates in real time due to factors such as equipment status and environmental changes. For example, passenger flow videos may become blurry due to crowd obstruction or changes in lighting; gate statistics may be missing or abnormal due to equipment malfunctions; and externally related data may lose timeliness due to information update delays. When the data quality of one or more modalities deteriorates significantly, the existing fixed-weight fusion method cannot automatically adapt to this dynamic change, resulting in an excessively high proportion of low-quality data in the fusion result, causing distortion of fusion features, and consequently affecting the accuracy of subsequent passenger flow predictions and the reliability of control decisions.

[0004] Therefore, how to adaptively adjust the fusion weights of various modalities under the condition of dynamic changes in the quality of multimodal data in subway stations, so as to ensure the robustness of passenger flow prediction and control, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a subway passenger flow prediction and control method, storage medium and electronic device, which solves the technical problem of low accuracy of passenger flow prediction in the prior art.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0009] In a first aspect, embodiments of the present invention provide a method for predicting and controlling subway passenger flow, comprising: collecting multimodal data of subway stations; wherein the multimodal data includes passenger flow video data; determining the quality factor of each data contained in the multimodal data, and performing gated fusion on all data contained in the multimodal data according to each determined quality factor to generate a multimodal fusion feature vector; performing panoramic stitching based on passenger flow video data to obtain a panoramic video image of the entire station, and using pedestrian kernel density estimation to map each pedestrian in the panoramic video image of the entire station to a Gaussian distribution and overlaying all Gaussian distributions. The system adds data to obtain the superimposed result, and corrects the superimposed result using occlusion adaptive weights determined by the video quality factor corresponding to the passenger flow video data. It then performs density adaptive Gaussian smoothing interpolation to generate a pixel-level passenger flow density heatmap. The multimodal fusion feature vector of the historical time series is input into the pre-built time series prediction model to obtain the predicted total passenger flow and the predicted regional passenger flow density distribution for future times. Based on the pixel-level passenger flow density heatmap, the predicted total passenger flow, and the predicted regional passenger flow density distribution, the station passenger flow control level is determined, and a control plan corresponding to the station passenger flow control level is generated.

[0010] Optionally, the multimodal data also includes gate passenger flow statistics and external data associated with passenger flow; wherein, the passenger flow data quality factor of the gate passenger flow statistics is determined based on the ratio of the number of effective sampling points to the theoretical number of sampling points and the proportion of outliers in the effective sampling data; the external data quality factor of the external data is determined based on the data update delay time of the external data; and the video quality factor is determined based on the image occlusion rate and image blur of the passenger flow video data.

[0011] Optionally, based on each determined quality factor, gated fusion is performed on all data contained in the multimodal data to generate a multimodal fusion feature vector: the dynamic fusion weights corresponding to each data in the multimodal data are determined according to the proportion of each quality factor in the sum of all quality factors; cross-modal attention is calculated on the feature vectors corresponding to each data in the multimodal data to obtain the gated fusion weights corresponding to each data in the multimodal data; the dynamic fusion weights corresponding to each data in the multimodal data are multiplied by the gated fusion weights to obtain a scaling weight vector; the scaling weight vector is then multiplied element-wise by the feature vector corresponding to that data to obtain the product result corresponding to each data in the multimodal data; and the product results corresponding to all data in the multimodal data are summed to obtain the multimodal fusion feature vector.

[0012] Optionally, passenger flow video data is collected through cameras; panoramic stitching is performed based on passenger flow video data to obtain a panoramic video image of the entire station, including: establishing a three-dimensional spatial constraint model to determine the effective shooting area of ​​each camera based on the field of view and deployment parameters of each camera; determining adjacent cameras based on the overlap of the effective shooting areas, extracting SIFT feature points and ORB feature points from the video frames collected by adjacent cameras respectively, and fusing the two types of feature points to obtain the feature point set corresponding to each video frame; performing feature point matching on the feature point sets of the video frames collected by adjacent cameras, and solving the homography matrix between adjacent video frames based on the matching results, and mapping, aligning and fusing the video frames collected by each camera participating in panoramic stitching based on the homography matrix to obtain a panoramic video image of the entire station.

[0013] Optionally, pedestrian kernel density estimation is used to map each pedestrian in the panoramic video of the entire station to a Gaussian distribution and then superimpose all Gaussian distributions to obtain the superimposed result. This includes: performing pedestrian target detection on the panoramic video of the entire station to obtain the center coordinates and size of each pedestrian; mapping each pedestrian to a Gaussian distribution with the center coordinates of each pedestrian as the center, and weighting and summing the values ​​of the Gaussian distributions of all pedestrians at each pixel in the panoramic video of the entire station with their respective pedestrian sizes as weights to obtain the basic passenger flow density of each pixel; after traversing all pixels, the basic passenger flow density of each pixel is used to construct an initial passenger flow density map, and the initial passenger flow density map is used as the superimposed result.

[0014] Optionally, the overlay result is corrected using an occlusion adaptive weight determined by the video quality factor corresponding to the passenger flow video data, and density adaptive Gaussian smoothing interpolation is performed to generate a pixel-level passenger flow density heatmap. This includes: determining the occlusion adaptive weight based on the video quality factor; wherein the occlusion adaptive weight increases as the occlusion degree represented by the video quality factor increases; correcting the overlay result using the occlusion adaptive weight to obtain a corrected passenger flow density map; and performing density adaptive Gaussian smoothing interpolation on the corrected passenger flow density map to generate a pixel-level passenger flow density heatmap; wherein the bandwidth of the Gaussian smoothing kernel used for density adaptive Gaussian smoothing interpolation is dynamically adjusted according to the local passenger flow density at the smoothed pixel.

[0015] Optionally, the temporal prediction model includes a modality-aware temporal coding layer, a gated attention encoder layer, and a spatiotemporal dual-branch decoder layer;

[0016] The modality-aware temporal coding layer is used to inject temporal position codes and modal feature codes into each multimodal fusion feature vector of the historical time series; wherein, the modal feature codes are obtained by concatenating the scaling weight vectors corresponding to each data in the multimodal data and then performing a linear transformation;

[0017] The gated attention encoder layer is used to encode the multimodal fusion feature vectors after the injection temporal position encoding and modal feature encoding using a gated attention mechanism. In the gated attention mechanism, the gated vector generated by the quality factors corresponding to each data in the multimodal data is multiplied element-wise with the standard Transformer attention output.

[0018] The spatiotemporal dual-branch decoder layer is used to predict the total passenger flow and the regional passenger flow density distribution respectively from the encoded features output by the gated attention encoder layer, and outputs the predicted total passenger flow and the predicted regional passenger flow density distribution.

[0019] Optionally, based on the pixel-level passenger flow density heatmap, the predicted total passenger flow, and the predicted regional passenger flow density distribution, the station passenger flow control level is jointly determined, and a control plan corresponding to the station passenger flow control level is generated. This includes: obtaining real-time passenger flow density from the pixel-level passenger flow density heatmap, calculating the passenger flow increase during the predicted period based on the predicted total passenger flow, and determining the congestion area distribution based on the predicted regional passenger flow density distribution; comparing the real-time passenger flow density, passenger flow increase, and preset abnormal event triggering signals with preset level-based judgment conditions; wherein each level-based judgment condition defines a corresponding density threshold condition, increase threshold condition, and abnormal event triggering condition; when at least two judgment conditions corresponding to the corresponding level are met, the corresponding level is determined as the station passenger flow control level; and based on the determined station passenger flow control level and congestion area distribution, a pre-built level-based control strategy knowledge base is retrieved to generate a control plan.

[0020] In a second aspect, embodiments of the present invention provide a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the subway passenger flow prediction and control method as described in any of the first aspects.

[0021] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0022] At least one processor;

[0023] And, a memory communicatively connected to the at least one processor;

[0024] The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, it causes the at least one processor to perform the subway passenger flow prediction and control method as described in any of the first aspects.

[0025] (III) Beneficial Effects

[0026] The beneficial effects of this invention are:

[0027] This application provides a method, storage medium, and electronic device for predicting and controlling subway passenger flow. It collects multimodal data from subway stations, including passenger flow video data. For each piece of collected multimodal data, a quality factor is determined. Based on each determined quality factor, gating fusion is performed on all data in the multimodal data to generate a multimodal fusion feature vector. When the quality of a piece of data deteriorates due to changes in equipment status or environment, its corresponding quality factor decreases, automatically reducing its contribution to the gating fusion and increasing the contribution of other high-reliability data. This filters out noise interference from low-quality data at the source, ensuring the accuracy of the fused features. Based on this, panoramic video stitching is performed using passenger flow video data to obtain a full-station panoramic video image. Pedestrian kernel density estimation is then used to map each pedestrian onto a Gaussian distribution, and the images are superimposed to obtain the superimposed result. The superimposed result is then corrected using occlusion adaptive weights determined by the video quality factor, and density adaptive Gaussian smoothing interpolation is performed to generate a pixel-level passenger flow density heatmap. Simultaneously, the multimodal fusion feature vector from historical time series is input into a pre-constructed time series prediction model to obtain the predicted total passenger flow and the predicted regional passenger flow density distribution for future times. Since the multimodal fusion feature vector has been adaptively adjusted for the contribution of each data point through the quality factor, the input to the time series prediction model possesses high robustness, thus ensuring the accuracy of the prediction results. Finally, based on the pixel-level passenger flow density heatmap, the predicted total passenger flow, and the predicted regional passenger flow density distribution, the station's passenger flow control level is jointly determined, and a corresponding control plan is generated, achieving end-to-end robustness assurance from data fusion to passenger flow prediction to control decision-making. Attached Figure Description

[0028] Figure 1 A flowchart of a subway passenger flow prediction and control method provided in an embodiment of this application is shown;

[0029] Figure 2 A flowchart illustrating a subway passenger flow prediction and control method provided in an embodiment of this application is shown. Detailed Implementation

[0030] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] The networked operation of urban rail transit has led to a continuous increase in passenger flow at subway stations. The safety risks such as overcrowding and stampedes, as well as the low operational efficiency caused by large passenger flows, are becoming increasingly prominent, placing extremely high demands on the real-time, accurate, and intelligent management of large passenger flows. While existing subway station large passenger flow management technologies have achieved passenger flow data collection and basic prediction, the following shortcomings still exist in core areas such as multimodal data fusion, short-term passenger flow prediction, and contingency plan generation:

[0032] First, existing technologies typically employ a fixed-weight approach to fuse multimodal data, including video, turnstile, and external data. However, in actual subway station operations, the quality of different modalities fluctuates in real time due to environmental changes: cameras are easily obstructed by crowds, and changes in lighting can cause blurry images; turnstile data often exhibits anomalies such as zero values ​​or jumps due to equipment malfunctions; and external data such as weather and events are subject to update delays. When the quality of a particular modality of data significantly deteriorates, the fixed-weight fusion method cannot automatically reduce the contribution of that modality to the fusion result, leading to distorted fusion features and insufficient accuracy in passenger flow perception and relevance to the scene.

[0033] Secondly, existing short-term passenger flow forecasting models mostly rely on single passenger flow time-series data, simply incorporating a small number of spatiotemporal features, and failing to fully cover multi-dimensional exogenous variables such as weather, holidays, and large-scale surrounding events, thus failing to capture the non-linear correlation between various factors and passenger flow. At the same time, traditional time-series models suffer from long-series dependency forgetting and insufficient exogenous variable modeling mechanisms, resulting in low accuracy in passenger flow forecasting at short timescales of 15 / 30 minutes, making it difficult to support precise control decisions.

[0034] Furthermore, existing control plans are fixed, manually preset schemes, only capable of "one plan per station," and cannot automatically generate targeted control plans based on real-time information such as predicted passenger flow levels and congested areas. Plan formulation relies on empirical rules and lacks integration with model reasoning and dynamic knowledge bases, making it difficult to achieve integrated decision-making for equipment scheduling, personnel allocation, and passenger guidance. In addition, there is a lack of feedback and iteration mechanisms after plan implementation, resulting in insufficient coordination, implementability, and continuous optimization capabilities of control measures, and a high degree of reliance on manual intervention.

[0035] In summary, existing technologies cannot achieve intelligent management and control of large passenger flows in subway stations across the entire process of "perception-prediction-decision-execution-feedback". There is an urgent need for a subway passenger flow prediction and control method to solve the technical problems of inaccurate passenger flow perception, low prediction accuracy, and rigid contingency plan generation.

[0036] Based on this, embodiments of this application provide a subway passenger flow prediction and control method, storage medium, and electronic device. The method involves collecting multimodal data from subway stations, including passenger flow video data. For each piece of collected multimodal data, a quality factor is determined. Based on each determined quality factor, gated fusion is performed on all data in the multimodal data to generate a multimodal fusion feature vector. When the quality of a piece of data deteriorates due to changes in equipment status or environment, its corresponding quality factor decreases, automatically reducing its contribution to the gated fusion and increasing the contribution of other high-reliability data. This filters out noise interference from low-quality data at the source, ensuring the accuracy of the fused features. Based on this, panoramic video stitching is performed using passenger flow video data to obtain a full-station panoramic video image. Pedestrian kernel density estimation is then used to map each pedestrian onto a Gaussian distribution, and the images are superimposed to obtain the superimposed result. The superimposed result is then corrected using occlusion adaptive weights determined by the video quality factor, and density adaptive Gaussian smoothing interpolation is performed to generate a pixel-level passenger flow density heatmap. Simultaneously, the multimodal fusion feature vector from historical time series is input into a pre-constructed time series prediction model to obtain the predicted total passenger flow and the predicted regional passenger flow density distribution for future times. Since the multimodal fusion feature vector has been adaptively adjusted for the contribution of each data point through the quality factor, the input to the time series prediction model possesses high robustness, thus ensuring the accuracy of the prediction results. Finally, based on the pixel-level passenger flow density heatmap, the predicted total passenger flow, and the predicted regional passenger flow density distribution, the station's passenger flow control level is jointly determined, and a corresponding control plan is generated, achieving end-to-end robustness assurance from data fusion to passenger flow prediction to control decision-making.

[0037] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0038] To facilitate understanding of the embodiments of this application, the terms involved in this application are explained below:

[0039] Spatiotemporal attention fusion: a feature fusion technology that simultaneously mines the temporal correlation (short-term time series, historical same period) and spatial correlation (station hall-platform-passage area association) features of subway passenger flow data;

[0040] Adaptive feature fusion: Dynamically allocate fusion weights based on the real-time quality of multimodal data to achieve accurate alignment between video semantic features and passenger flow statistics features;

[0041] Gated fusion: It is a feature fusion mechanism that uses learnable gating vectors to control the contribution intensity of different modal features during fusion on an element-by-element basis, thereby achieving adaptive selection and weighted combination of modal features;

[0042] Tiered response mechanism: Based on the predicted large passenger flow level (Level 1 / Level 2 / Level 3), different levels of passenger flow control emergency measures will be automatically triggered.

[0043] MTS-Former (Multi-modal Time-Series Transformer): The proposed time series prediction model for subway scenarios takes multimodal fusion features as input and achieves joint and accurate prediction of total passenger flow and regional heat map through three major modules: modality-aware encoding, adaptive gating attention, and spatiotemporal dual-branch decoding.

[0044] Kernel Density Estimation (KDE): It is a non-parametric density estimation method. In this application, it is used to map each pedestrian to a Gaussian distribution and generate a pixel-level passenger flow density heatmap by superimposing the Gaussian distributions of all pedestrians.

[0045] Homography Matrix: It is a 3×3 matrix that describes the projection transformation relationship between two planes. In this application, it is used to achieve precise alignment and stitching of pixel coordinates between adjacent camera images.

[0046] Please see Figure 1 , Figure 1 A flowchart illustrating a subway passenger flow prediction and control method provided in an embodiment of this application is shown. Figure 1 As shown, the subway passenger flow prediction and control method can be executed by electronic equipment, and the specific device of the electronic equipment can be configured according to actual needs; the embodiments of this application are not limited thereto. For example, the electronic equipment can be a computer or a server, etc. Specifically, the subway passenger flow prediction and control method includes:

[0047] Step S110: Collect multimodal data of the subway station. This multimodal data includes at least passenger flow video data.

[0048] It should be understood that the data contained in the multimodal data can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0049] Optionally, multimodal data can be collected from key areas of the subway station (such as the concourse, platform, transfer passages, and entrances / exits). This multimodal data may include passenger flow video data, turnstile passenger flow statistics, and external data related to passenger flow. Passenger flow video data may come from cameras deployed in the aforementioned areas; turnstile passenger flow statistics can provide statistical information such as the number of people entering and exiting the station and instantaneous cross-sectional flow velocity. Furthermore, external data surrounding the station can be obtained in real time through the city's government affairs platform. This external data may include weather data (such as temperature and rainfall) and holiday / weekday tags, and information on large-scale events in the vicinity, including event type, event time, and estimated end time, can be simultaneously accessed.

[0050] Step S120: Determine the quality factor of each data in the multimodal data, and perform gated fusion on all data in the multimodal data according to each determined quality factor to generate a multimodal fusion feature vector.

[0051] It should be understood that the specific process for determining the quality factor of each data point contained in the multimodal data can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0052] Optionally, the passenger flow data quality factor of the gate passenger flow statistics is determined based on the ratio of the number of effective sampling points to the theoretical number of sampling points and the proportion of outliers in the effective sampling data; the external data quality factor of the external data is determined based on the data update delay time of the external data; and the video quality factor is determined based on the screen occlusion rate and screen blur of the passenger flow video data.

[0053] Specifically, the processing of camera video data mainly involves two parts: First, the video streams captured by subway station cameras are deblurred, denoised, and invalid frames are removed to ensure the clarity and validity of each frame; second, panoramic stitching and deduplication of the entire station's camera footage are performed to obtain a 1:1 panoramic video image corresponding to the station's physical space. For details, please refer to [link to relevant documentation]. Figure 2 Related descriptions.

[0054] (2.1) Non-blind defuzzing method based on kernel estimation

[0055] In subway stations, dense passenger flow, rapid pedestrian movement, and network lag can easily cause motion blur in videos. This application employs a non-blind deblurring algorithm based on gradient domain kernel estimation, which first estimates the blur kernel and then performs image restoration. The formula derivation and implementation steps are as follows:

[0056] Let the blurred video frame be I. blur Clear video frames are I clear The motion fuzzy kernel is k, and the mathematical model of the fuzzy process is:

[0057] ;

[0058] In the formula, represents a two-dimensional convolution operation; n represents additive noise.

[0059] The solution for the blur kernel is based on the core principle of the sparsity of the image gradient domain. Therefore, this scheme calculates the image gradient domain... Based on the sparsity, construct an optimization objective function. Solving for the fuzzy kernel:

[0060] ;

[0061] In the formula, This represents the estimated values ​​of the blur kernel k and the sharp frame. Find the minimum value; The estimated value of the sharp frame (which gradually approximates the true sharp frame I during the algorithm iteration process). clear ); Represents the sharp frame estimate Image gradient (representing changes in the edges and details of the image); The L1 norm of the gradient; This represents the regularization coefficient (ranging from 0.01 to 0.05). The square of the L2 norm of the restoration error; This indicates that the constraints are met; The L1 norm normalization of the fuzzy kernel k is represented; Non-negativity constraint of fuzzy kernel k.

[0062] Based on the blur kernel k obtained in the previous step, the video frames are deblurred through deconvolution. To avoid ringing artifacts during deconvolution, a Gaussian smoothing term G is introduced.

[0063] ;

[0064] In the formula, This represents the calculation of Gaussian convolution; This indicates deconvolution calculation; the standard deviation of the Gaussian kernel G is taken as 0.5~1.0.

[0065] Denoising method based on adaptive Gaussian bilateral filtering:

[0066] To address Gaussian noise and salt-and-pepper noise in videos, an adaptive Gaussian bilateral filter is employed, balancing noise removal with image edge preservation. This avoids the over-smoothing problem associated with traditional filters. The implementation method is as follows:

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] In the formula, This represents the gray value of the denoised image at pixel coordinates (x, y); Indicates the normalized weighting factor; Represents the neighborhood window The coordinates of any pixel within the range; It is a neighborhood window of the pixel (x,y) (usually 3×3 or 5×5); This represents the original blurred (noisy) image in pixel coordinates. The grayscale value at that location; Indicates the spatial domain weight; Indicates the weight of the grayscale area; Represents an exponential function; Take 1~2; It can adaptively adjust based on the local variance of the image. Here, k takes values ​​between 0.1 and 0.2, and Represents the local variance of the neighborhood; It represents the square of the grayscale difference between the center pixel and its neighboring pixels.

[0072] Invalid frame processing method based on multi-feature threshold determination:

[0073] This application establishes three core judgment features. When a video frame meets any invalid condition, it is directly discarded to ensure that only valid frames enter the subsequent splicing process. The judgment rules are as follows:

[0074] Feature 1: Average Frame Brightness: Calculate the average grayscale brightness L of the frame. (Black screen) 20 can be selected) or (overexposure, (240) is determined to be an invalid frame;

[0075] Feature 2: Pixel Variance: Calculates the variance of pixel grayscale values ​​in a frame. ,like (The image lacks detail, resembling a solid color frame.) If the value is 10, it is determined to be an invalid frame;

[0076] Feature 3: Camera status indicator: If the device interface reports that the camera is offline or malfunctioning, the corresponding frame is directly determined to be an invalid frame.

[0077] (2.2) Passenger flow data preprocessing:

[0078] The passenger flow data from the turnstiles includes the instantaneous and cumulative number of people entering / exiting the station. This data is prone to anomalies such as zero values, extreme values, and jump values ​​due to equipment malfunctions (e.g., turnstile jamming, signal interruption). Preprocessing follows the principle of "removing anomalies first, then segmenting by region" to ensure the accuracy of the passenger flow data.

[0079] This application directly identifies negative values ​​in the data as outliers, and for other situations, it adopts... The principle combines a two-layer elimination strategy based on sliding window trend judgment to both eliminate obvious statistical outliers and avoid mistakenly deleting normal peak values ​​caused by large passenger flows. The specific implementation method is as follows:

[0080] Initial screening for outliers:

[0081] Suppose the time series of passenger flow data for a certain turnstile is: (t is the timestamp, and the sampling granularity is 1 minute), calculate the mean of the data. and standard deviation :

[0082] ;

[0083] ;

[0084] In the formula, T represents the total number of data points in the data sequence; This represents the value of the t-th data point in the time series.

[0085] And, set an anomaly detection threshold: if If it is initially judged to be an outlier, it will be marked as such. .

[0086] Sliding window trend determination and rescreening:

[0087] For the initial screening outliers in the previous step A sliding window (window size W=5) is used to analyze the trend of the data before and after the peak, avoiding misjudging normal peaks caused by large passenger flows as outliers. The judgment rules are as follows:

[0088] Calculate the rate of change of passenger flow within the sliding window This reflects data trends;

[0089] like ( (Taking 30 people / minute, i.e., the data jumps too much), and and Deviation rate of (mean of data within the window) If it is an outlier, it will be identified as a real outlier and removed.

[0090] If the above conditions are not met, the large passenger flow is considered normal and the passenger flow will be retained.

[0091] Outlier completion:

[0092] For outliers that have been removed, linear interpolation is used to complete the data, ensuring data continuity.

[0093] .

[0094] (2.3) External data preprocessing

[0095] External data includes weather data (temperature, rainfall), holiday / workday labels, and surrounding activity data (activity type, scale, event time, and end-of-event information). These are divided into discrete and continuous variables, which need to be coded and normalized separately. Non-numerical variables are converted into numerical variables, and continuous variables are mapped to a unified interval to meet the numerical input requirements of the prediction model and eliminate the influence of differences in units on the model results.

[0096] Discrete variables:

[0097] For discrete variables such as holiday type (weekday / weekend / statutory holiday), event type (concert / sports event / exhibition), and weather condition (sunny / rainy / snowy), one-hot encoding is used to quantify the categorical features, thus avoiding the model from misclassifying the categorical features as ordered features.

[0098] Let a discrete variable be... It contains m categories, and for each category Generate a binary vector of length m. :

[0099] ;

[0100] In the formula, i represents the index of the current category in the category set; j represents the index of the vector dimension, and For vector dimensions.

[0101] For example, holiday types (m=3), then working days ,weekend statutory holidays .

[0102] Continuous variables:

[0103] For continuous variables such as temperature (°C), rainfall (mm), and event size (number of participants), deviation standardization is used to map them to the [0,1] interval to eliminate dimensional differences (e.g., temperature is in °C, rainfall is in mm). The implementation method is as follows:

[0104] ;

[0105] In the formula, Represents the normalized variable value ( x is the original value of a continuous variable; and These are the minimum and maximum values ​​of the variable in the historical dataset.

[0106] In the actual processing, it was found that the proportion of zero rainfall values ​​was too high. A large number of zero values ​​would lead to an imbalance in the distribution of standardized data. Therefore, based on this situation, further logarithmic transformation was performed on the data before data standardization.

[0107] ;

[0108] In the formula, x+1 is used to avoid the meaninglessness caused by zero values ​​in logarithmic operations.

[0109] See also Figure 2 After completing the preprocessing and standardization of the multi-source data (video, passenger flow, and external data), although the data from different modalities are unified in numerical form, they still exist in their own independent coordinate systems, lacking a unified spatiotemporal reference and mutual reliability metrics. To effectively fuse heterogeneous data, a cross-modal feature alignment mechanism needs to be established. This mechanism first performs strong spatiotemporal alignment on the three types of data, and then evaluates the data quality for each modality in real time, providing a foundation for subsequent adaptive fusion.

[0110] (2.4) Spatiotemporal Dimension Alignment

[0111] All preprocessed modal data first undergoes strong spatiotemporal alignment: using the BeiDou clock of the subway station operation and control platform as a unified time reference (time format: YYYY-MM-DD HH:MM:SS.fff, where fff represents milliseconds, e.g., 2026-04-26 14:35:09.123 represents 14:35:09.123 on April 26, 2026), clock synchronization is performed on all acquisition devices to ensure that the device clock synchronization error is ≤10ms. After alignment, a time index table is built for all timestamped data, supporting fast retrieval and matching of multi-source data by timestamp. Based on the station CAD drawings and panoramic video images, the station is divided into different areas, and each area is assigned an area code ID as a unique identifier after grid division, binding a unified spatial stamp (area code ID) to all spatially related data.

[0112] (2.5) Multidimensional data quality assessment factors

[0113] Calculate the real-time data quality factor for each type of modality. This characterizes the reliability of the data at the current moment:

[0114] Video data quality factor:

[0115] Video data quality factor Used to measure image sharpness, occlusion rate, and the proportion of effective information. When the camera is obstructed and the image is blurry... Automatic descent reduces the proportion of video features in the fusion process, avoiding misinformation from interfering with prediction.

[0116] ;

[0117] In the formula, The screen occlusion rate (0~1); Image blur level (0~1); Indicates the weighting coefficients of the two (e.g.) =0.7、 =0.3).

[0118] Passenger flow data quality factor:

[0119] Passenger flow data quality factor Used to measure the integrity, anomaly rate, and volatility of gate data. This is applied when there is a gate malfunction, network outage, or data loss. Automatic descent avoids prediction errors caused by distorted passenger flow statistics.

[0120] ;

[0121] In the formula, The number of valid sampling points; This represents the total number of theoretical sampling points. For abnormal penalty coefficients (e.g.) =2); E represents the percentage of outliers in the valid sampled data.

[0122] External data quality factor:

[0123] External data quality factor Used to measure the timeliness and effectiveness of updates to weather, events, and time tags. External data is automatically downweighted when it becomes outdated or invalid, ensuring that forecasts are always based on the latest real-world conditions.

[0124] ;

[0125] In the formula, The time decay coefficient (e.g.) =0.02 / min); Data update delay time (in minutes).

[0126] It should also be understood that the specific process of gating and fusing all data contained in the multimodal data according to each determined quality factor to generate the multimodal fusion feature vector can also be set according to actual needs, and the embodiments of this application are not limited thereto.

[0127] Optionally, based on the proportion of each quality factor in the sum of all quality factors, the dynamic fusion weights corresponding to each data point in the multimodal data are determined; cross-modal attention is calculated on the feature vectors corresponding to each data point in the multimodal data to obtain the gated fusion weights corresponding to each data point in the multimodal data; the dynamic fusion weights corresponding to each data point in the multimodal data are multiplied by the gated fusion weights to obtain the scaling weight vector; the scaling weight vector is then multiplied element-wise by the feature vector corresponding to that data point to obtain the product result corresponding to each data point in the multimodal data; and the product results corresponding to all data points in the multimodal data are summed to obtain the multimodal fusion feature vector.

[0128] Specifically, the three types of data quality factors defined based on the above content. , , It automatically calculates the fusion weights of each modality's features, and the fusion weights are adjusted in real time according to data quality. When the data quality of a certain modality deteriorates, its fusion weight is automatically reduced, and other high-confidence modalities automatically fill the gap, achieving modality complementarity and intelligent fault tolerance.

[0129] ;

[0130] ;

[0131] ;

[0132] In the formula, Indicates video data quality factor The corresponding dynamic fusion weights; Indicates passenger flow data quality factor The corresponding dynamic fusion weights; Represents external data quality factor The corresponding dynamic fusion weights. Furthermore, the above must satisfy: ; .

[0133] The adaptive dynamic weight allocation mechanism in this solution possesses intelligent robustness that traditional fixed fusion methods completely lack: when cameras are functioning normally, passenger flow is normal, and there are no special events, the three factors are balanced and fused; when cameras are severely obstructed, the weight allocation is automatically reduced. Increase the weight of passenger flow and external data; automatically reduce the weight when the gate data is abnormal. Increase the weight of video and external data; automatically increase the weight when there is extreme weather or large-scale event near the station. This strengthens the impact of external events on passenger flow.

[0134] Furthermore, after achieving adaptive dynamic weight allocation for data from different modalities, this scheme first constructs a modality-specific feature embedding layer to uniformly map the three types of heterogeneous features into high-dimensional feature vectors of the same dimension; then, it performs gated fusion of the cross-modal features of the same dimension to generate a multimodal fusion feature vector F.

[0135] ;

[0136] In the formula, These represent the video semantic feature vector, the dynamic weight of video data quality, and the gating fusion weight of video semantic features, respectively. These represent the passenger flow statistics feature vector, the dynamic weight of passenger flow data quality, and the passenger flow statistics feature gating fusion weight, respectively. These represent the external correlation feature vector, the external data quality dynamic weight, and the external correlation feature gating fusion weight, respectively.

[0137] The three types of features mentioned above have completely different original input dimensions and data types (videos use visual semantics, passenger flow uses statistical values, and external features use event labels). Dimensional unification is achieved through an independently trainable feature embedding layer (modality-specific encoder). The core logic of cross-modality feature alignment is: each modality corresponds to a dedicated embedding network, which learns the mapping relationship between the original features of that modality and a high-dimensional space, preserving modality-specific semantics; the output dimension of the embedding layer is forced to be unified to D, regardless of the original input dimension, ultimately mapping to a vector with the same structure; the embedding layer parameters are optimized through backpropagation during model training to ensure that the mapped vector can effectively represent the core information of that modality, avoiding information loss caused by dimensional unification.

[0138] This application sets a uniform feature dimension D=128, and the embedding layers of the three types of features all use D as the target output dimension, structurally ensuring... All are 1×128 row vectors, satisfying the mathematical condition for direct weighted addition.

[0139] as well as, The dynamic fusion weights for the three types of features are scalars calculated from the data quality factor, representing the overall data credibility of each feature.

[0140] as well as, The gating fusion weights for the three types of features are 1×D dimensional vectors generated by the cross-modal attention mechanism, representing the dynamic importance of each feature dimension.

[0141] The fusion mechanism of this application breaks through the limitations of the traditional fixed weight simple addition. Through the dual weighting of global quality constraints and local feature attention, a strong intrinsic relationship between video, passenger flow and external events is established: video represents spatial form, passenger flow represents numerical intensity, and external representation represents driving causes. The three mutually verify and correct each other, providing a highly reliable and interpretable fusion feature foundation for the subsequent generation of short-term passenger flow prediction and control plans.

[0142] It should be noted here that the above video quality factors Passenger flow data quality factor and external data quality factors The data is calculated and cached in real time for adaptive gating fusion in subsequent step S120. However, in actual subway operation scenarios, when the quality factor of a certain modality drops to an extremely low level (e.g., video quality factor below 0.15 or passenger flow data quality factor below 0.15), the effective information of that modality is severely lacking. Simply reducing its dynamic fusion weight in gating fusion can only reduce its negative impact but cannot recover the information loss, leading to a decrease in the information completeness of the fused features. To compensate for this deficiency, this scheme further introduces the following cross-modal knowledge distillation and feature repair steps before entering gating fusion.

[0143] Specifically, the quality factor of each modality in the multimodal data is acquired in real time; the quality factor of each modality is compared with a preset high-quality threshold and a low-quality threshold to determine whether there is a teacher-student modality pair where the quality factor of the teacher modality is higher than the high-quality threshold and the quality factor of the student modality is lower than the low-quality threshold; if such a teacher-student modality pair exists, a corresponding strategy is selected from a variety of preset knowledge distillation strategies according to the specific types of the teacher and student modalities, and distillation features for repairing the student modality are generated using the feature information of the teacher modality; the distillation features are fused with the original collected features of the student modality using confidence weighting to obtain the repaired student modality features, wherein the fusion weight is determined by the product of the quality factor of the teacher modality and the distillation confidence; all repaired modality features are sent to the subsequent adaptive gating fusion step to generate a multimodal fusion feature vector. The technical advantages of the above method are as follows: from passive harm avoidance to active repair, it utilizes high-quality modal active distillation knowledge to repair low-quality modalities, fundamentally ensuring the information completeness of fused features; when the quality factor of a certain modality is extremely low, reliable alternative features can still be generated through cross-modal knowledge transfer, ensuring the system's continued usability in extremely harsh data environments; dedicated distillation strategies are designed for three scenarios: video to gate passenger flow, gate passenger flow to video, and fused modality to external data, resulting in more accurate and scenario-adaptable features after repair; and the integration degree of repaired features is dynamically controlled by the product of teacher quality factor and distillation confidence, achieving an intelligent balance between repairability and conservatism.

[0144] For example, firstly, the system configures a mutual learning trigger gating for each modal combination (passenger flow video data and gate passenger flow statistics, passenger flow video data and external data, gate passenger flow statistics and external data). This gating determines whether to initiate the knowledge distillation process based on the relationship between the quality factor of each modality at the current moment and a preset threshold.

[0145] If the quality factor of one modality is higher than the preset high quality threshold (e.g., 0.85) and the quality factor of another modality is lower than the preset low quality threshold (e.g., 0.15), then the former is selected as the "teacher modality" and the latter is selected as the "student modality" to be repaired, triggering knowledge distillation from teacher to student.

[0146] If multiple teacher-student modal pairs that meet the conditions exist simultaneously, the system retains only the teacher modality with the highest quality factor to avoid multi-source knowledge conflicts.

[0147] If no teacher-student modal pair meets the conditions, skip this step and proceed directly to the gating fusion in step S120.

[0148] Secondly, based on the specific types of teacher and student modalities, the system implements the following three differentiated knowledge distillation strategies:

[0149] Strategy 1: Knowledge distillation from passenger flow video data (teachers) to gate passenger flow statistics (students).

[0150] Applicable conditions: The video quality factor is higher than the preset high quality threshold, and the passenger flow data quality factor is lower than the preset low quality threshold.

[0151] The implementation steps are as follows:

[0152] From the pixel-level passenger flow density heat map generated in step S130, extract the instantaneous passenger flow at each gate section.

[0153] A repair network based on a temporal attention mechanism is constructed. The input to this network is the original gate passenger flow data sequence within the historical window and the current passenger flow volume perceived by the video. The output is the repaired current gate passenger flow data. The specific network of this repair network can be configured according to actual needs, and this embodiment is not limited to this. For example, the repair network can employ any one of the following: a Transformer temporal repair network, an LSTM-attention hybrid network, a temporal convolutional network, or a lightweight gated recurrent unit.

[0154] The repaired gate passenger flow data replaces the original low-quality gate passenger flow data and is used as the distillation feature of this student mode.

[0155] Strategy 2: Knowledge distillation from gate passenger flow statistics (teachers) to passenger flow video data (students).

[0156] Applicable conditions: Passenger flow data quality factor is higher than the preset high quality threshold, and video quality factor is lower than the preset low quality threshold.

[0157] The implementation steps are as follows:

[0158] The total number of passengers at the current moment, as calculated by the gate passenger flow statistics, is used as the global monitoring signal.

[0159] A density map corrector is constructed, taking the initial passenger flow density map generated by pedestrian kernel density estimation in step S130 as input and outputting the corrected passenger flow density heatmap. This corrector uses a differentiable spatial allocation network to softly distribute the total passenger flow to each pixel location, ensuring that the total passenger flow in the corrected heatmap matches the total passenger flow statistics from the turnstiles.

[0160] The corrected passenger flow density heatmap replaces the original initial passenger flow density map as the distillation feature of the student mode, and is used for occlusion adaptive correction and heatmap generation in subsequent step S130.

[0161] Strategy 3: Knowledge distillation from fusion modalities (teachers) to external data (students).

[0162] Applicable conditions: The external data quality factor is lower than the preset low quality threshold, and the maximum value of the video quality factor and the passenger flow data quality factor is higher than the preset high quality threshold.

[0163] The implementation steps are as follows:

[0164] The query vector is either the gate passenger flow statistics sequence after Strategy 1 repair, or the station-wide passenger flow time series data obtained from pixel-level passenger flow density heatmap statistics after Strategy 2 repair. This query vector represents the current passenger flow change pattern over time.

[0165] Temporal similarity matching is performed in the pre-built "Event-Passenger Flow Response Pattern Knowledge Base" to retrieve the historical event pattern most similar to the current query vector. The Event-Passenger Flow Response Pattern Knowledge Base stores typical historical external events and their corresponding passenger flow temporal response patterns. Each record includes an event type label (e.g., large concerts, sporting events, extreme weather), the time characteristics of the event (e.g., the time of dispersal), and the standardized temporal waveform of station passenger flow under the influence of the event (the sampling interval is consistent with the historical window in step (140)). This knowledge base can be constructed offline: historical data from the same period are collected, the passenger flow temporal sequence on the day the external event occurs is clustered, typical waveforms of various events are extracted, and manual annotation is used to form a pattern library. Temporal similarity matching can be Dynamic Time Warping (DTW). By calculating the DTW distance between the query vector and each template in the knowledge base, the template with the smallest distance and less than a preset threshold (e.g., 0.2) is selected as the matching result, and its event type is the inferred result. The matching similarity (1 - normalized DTW distance) is used as the confidence level.

[0166] The output is an inferred external event feature vector, including event type, matching confidence, and expected duration of impact. This inferred feature serves as a distillation feature of the external data modality.

[0167] Furthermore, for each repaired student modality, the final output features are not directly derived from the repaired features generated by distillation, but rather are fused with the original acquired features using a confidence-weighted method. The fusion formula is as follows:

[0168] Final features = (quality factor of teacher modality × distillation confidence) × repaired features + (1 - quality factor of teacher modality × distillation confidence) × original collected features.

[0169] The distillation confidence is determined according to different strategies: in strategy one, the mean of the attention weights is taken; in strategy two, the maximum value of the output probability of the spatial allocation network is taken; and in strategy three, the similarity score of temporal matching (i.e., 1 - normalized DTW distance) is taken.

[0170] When the quality factor of the teacher modality is high and the distillation confidence is high, the repair features dominate; otherwise, some original information is retained to avoid introducing erroneous knowledge.

[0171] The repaired final modal features (including repaired passenger flow video data features, repaired gate passenger flow statistics data features, and repaired external data features) are uniformly sent to the gating fusion step in step S120 to generate a multimodal fusion feature vector.

[0172] Step S130: Based on passenger flow video data, panoramic stitching is performed to obtain a panoramic video image of the entire station. Pedestrian kernel density estimation is used to map each pedestrian in the panoramic video image of the entire station to a Gaussian distribution and all Gaussian distributions are superimposed to obtain the superimposed result. The superimposed result is corrected by occlusion adaptive weight determined by the video quality factor corresponding to the passenger flow video data, and density adaptive Gaussian smooth interpolation is performed to generate a pixel-level passenger flow density heat map.

[0173] It should be understood that the specific process of performing panoramic stitching based on passenger flow video data to obtain a panoramic video image of the entire station can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0174] Optionally, a three-dimensional spatial constraint model is established based on the field of view and deployment parameters of each camera to determine the effective shooting area of ​​each camera; adjacent cameras are identified based on the overlap of their effective shooting areas, and SIFT and ORB feature points are extracted from the video frames captured by adjacent cameras respectively. These two types of feature points are then fused to obtain the feature point set corresponding to each video frame; feature point matching is performed on the feature point sets of the video frames captured by adjacent cameras, and the homography matrix between adjacent video frames is solved based on the matching results. Finally, based on the homography matrix, the video frames captured by each camera participating in panoramic stitching are mapped, aligned, and fused to obtain the full-site panoramic video image. For details, please refer to [link to relevant documentation]. Figure 2 Related content.

[0175] Therefore, using the preprocessed single-camera effective video frames from step S120 as input, a four-step algorithm based on 3D spatial constraint modeling, multi-scale feature point matching, adaptive fusion of overlapping areas, and blind zone interpolation completion is used to generate a full-station panoramic video image that corresponds to the physical space of the subway station in a 1:1 ratio. This provides a unified pixel-level spatial base for the generation of dynamic heat maps and solves the problems of "regional fragmentation and spatial misalignment" in traditional single-camera heat maps.

[0176] Specifically, (3.1) Three-dimensional spatial constraint modeling

[0177] Based on the station's CAD drawings and camera equipment parameters, the two-dimensional projection of the effective shooting area of ​​a single camera is calculated, and a mapping relationship between "camera image - physical space" is established to avoid splicing misalignment.

[0178] The horizontal / vertical field of view (FOV) of a camera determines its shooting range in physical space, and the calculation method is as follows:

[0179] ;

[0180] ;

[0181] In the formula, The horizontal field of view represents the shooting angle in the left and right directions of the camera, which is determined by the width w and focal length f of the camera's image sensor. The vertical field of view represents the shooting angle in the left and right directions of the camera, which is determined by the height h of the camera's image sensor and the focal length f.

[0182] Furthermore, based on the field of view and installation angle, the projected polygon of the effective shooting area of ​​the camera on the ground is calculated. :

[0183] First, based on the camera installation location Starting from the horizontal deflection angle Vertical pitch angle Calculate the field-of-view boundary ray of the camera;

[0184] Secondly, the intersection of the boundary ray of the field of view and the ground is the projected polygon. The coordinates of the four vertices;

[0185] And polygons This refers to the effective shooting range of the camera in physical space, encompassing all cameras. Together covering the entire physical space of the station .

[0186] (3.2) Multi-scale feature point fusion matching

[0187] Adjacent camera feeds exhibit differences in viewpoint, scale, and rotation. Feature point matching is required to solve the homography matrix, achieving precise alignment of pixel coordinates and providing a coordinate basis for subsequent fusion and deduplication. Traditional single-feature-point matching is prone to omissions and mismatches. This solution employs SIFT+ORB multi-scale feature point fusion matching to improve matching accuracy and anti-interference capabilities.

[0188] Multi-scale feature point extraction:

[0189] SIFT feature point extraction: To address scale invariance, a Gaussian pyramid is constructed for the image, and extreme points are extracted in different scale spaces to generate 128-dimensional feature descriptors that adapt to image scaling and lighting changes.

[0190] ORB Feature Point Extraction: To address rotation invariance, based on FAST corner detection and BRIEF descriptors, a 256-dimensional feature descriptor is generated to adapt to screen rotation and viewpoint changes.

[0191] Subsequently, the two types of feature points mentioned above are combined to obtain the image. feature point set , and the picture Images from adjacent cameras feature point set This balances scale and rotation invariance. and This represents two video frames captured simultaneously by two adjacent cameras, where m represents the camera frame. The total number of feature points extracted, where n represents the number of camera images. The total number of feature points extracted; each feature point in the two feature point sets can contain the pixel coordinates and feature descriptor information of that point.

[0192] Feature point matching:

[0193] Using the K-nearest neighbor matching method (K=2), for Each feature point, in Find the two closest feature points and calculate their distance ratio. ( For the closest distance, (Second closest distance); set a threshold. (like =0.8), if If a pair is deemed a high-quality match, it is discarded; otherwise, it is discarded. Finally, cross-validation is used to select only the best matches. arrive , arrive Feature points for bidirectional matching.

[0194] Solving for the homography matrix:

[0195] homography matrix It is to achieve the picture arrive The core of pixel coordinate mapping is satisfying the homogeneous coordinate transformation relationship:

[0196] ;

[0197] In the formula, for Feature point pixel coordinates; for Feature point pixel coordinates; It contains 9 parameters, constrained by the scale invariance of homogeneous coordinates. To solve for the other 8 unknown parameters, at least 4 sets of matching points are needed.

[0198] For each set of matching points , Expanding the homogeneous transformation yields two linear equations:

[0199] ;

[0200] Substitution Arranged into a system of linear equations ,in:

[0201] ;

[0202] ;

[0203] .

[0204] Finally, based on the homography matrix The video frames captured by each camera participating in the panoramic stitching are mapped, aligned, and merged to obtain a panoramic video image of the entire site.

[0205] It should also be understood that the specific process of using pedestrian kernel density estimation to map each pedestrian in the panoramic video of the entire station to a Gaussian distribution and superimposing all Gaussian distributions to obtain the superimposed result can also be set according to actual needs, and the embodiments of this application are not limited thereto.

[0206] Optionally, pedestrian target detection is performed on the panoramic video of the entire site to obtain the center coordinates and size of each pedestrian; with the center coordinates of each pedestrian as the center, each pedestrian is mapped to a Gaussian distribution, and the values ​​of the Gaussian distribution of all pedestrians at each pixel of the panoramic video of the entire site are weighted and summed with their respective pedestrian sizes to obtain the basic passenger flow density of each pixel; after traversing all pixels, the basic passenger flow density of each pixel is used to form an initial passenger flow density map, and the initial passenger flow density map is used as the overlay result.

[0207] Specifically, see [link to relevant documentation] Figure 2 Traditional heatmaps count pedestrians using fixed grid units. This solution uses Kernel Density Estimation (KDE) to map each pedestrian to a Gaussian distribution. By overlaying the Gaussian distributions of all pedestrians, pixel-level pedestrian flow density is directly generated, completely replacing fixed grid statistics and preserving pixel-level details.

[0208] ;

[0209] ;

[0210] In the formula, is a Gaussian kernel function, representing the density contribution of a single pedestrian to surrounding pixels; the closer the distance to the pedestrian, the higher the contribution. These are the pixel coordinates in the panoramic stitched image, with each pixel corresponding to a unique physical spatial location. Let the center coordinates of the k-th pedestrian be _____. To adapt to bandwidth, it solves the problem of "overly dense foreground and overly sparse background" caused by traditional fixed bandwidth; Let the pedestrian dimensions be those of the k-th pedestrian. For the preset scaling factor (e.g.) =3).

[0211] For each pixel in the panoramic image The basic passenger flow density is obtained by superimposing the Gaussian kernel contributions of all pedestrians. (Unit: people / m) 2 ):

[0212] ;

[0213] In the formula, K represents the total number of pedestrians detected in the panoramic video frame. Additionally, pedestrian dimensions can also be used. As a weighting factor, the density contribution of large-sized (close-up) pedestrians is higher, while the contribution of small-sized (far-up) pedestrians is lower, which is consistent with the actual passenger flow distribution in the physical space.

[0214] The initial passenger flow density map can then be used as the overlay result.

[0215] It should also be understood that the specific process of correcting the superposition result by means of the occlusion adaptive weight determined by the video quality factor corresponding to the passenger flow video data and performing density adaptive Gaussian smooth interpolation to generate a pixel-level passenger flow density heat map can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0216] Optionally, an adaptive occlusion weight is determined based on the video quality factor; wherein the adaptive occlusion weight increases as the degree of occlusion represented by the video quality factor increases; the superposition result is corrected using the adaptive occlusion weight to obtain a corrected passenger flow density map; the corrected passenger flow density map is subjected to density-adaptive Gaussian smoothing interpolation to generate a pixel-level passenger flow density heatmap; wherein the bandwidth of the Gaussian smoothing kernel used for density-adaptive Gaussian smoothing interpolation is dynamically adjusted according to the local passenger flow density at the smoothed pixel.

[0217] Specifically, see [link to relevant documentation] Figure 2 To address the issue of missed target detection due to factors such as pedestrian overlap and object occlusion within the station, a video quality factor is introduced. It adaptively corrects the base density to compensate for density errors caused by missed detections.

[0218] Based on the video data quality factor in step S120, it is refined into pixel-level occlusion correction weights. :

[0219] ;

[0220] ;

[0221] In the formula, For pixels Video quality factor at location; O is occlusion rate; B is blur. The minimum value (taken as 10) -6 Avoid having a denominator of zero (i.e., avoid...) ).

[0222] Introducing corrected weights When the screen occlusion rate is higher ( When (smaller), The larger the value, the stronger the compensation for missed detections; no obstruction ( When =1), No additional corrections are required, ensuring the density accuracy of normal areas.

[0223] Multiplying the adjusted weights by the base density yields the passenger flow density after adaptive correction for occlusion:

[0224] .

[0225] Furthermore, to further enhance the visual continuity of the heatmap and eliminate the slight graininess caused by the superposition of pedestrian kernels, the corrected density map is subjected to full-map adaptive Gaussian smoothing interpolation. The smoothing kernel bandwidth is dynamically adjusted with density to avoid the loss of details caused by excessive smoothing.

[0226] First, a density-adaptive Gaussian smoothing kernel is used to optimize the smoothing effect. A larger bandwidth is used in high-density areas (crowded areas) to enhance the smoothing effect, while a smaller bandwidth is used in low-density areas to preserve details.

[0227] bandwidth From density Dynamic calculation:

[0228] ;

[0229] In the formula, For smoothing coefficients (e.g.) =0.5); The base bandwidth (1 pixel) is used to ensure that details in low-density areas are preserved.

[0230] For the corrected density map Perform adaptive Gaussian smoothing to obtain the final passenger flow density (or pixel-level passenger flow density heatmap). :

[0231] ;

[0232] ;

[0233] ;

[0234] In the formula, For pixels to be smoothed The neighboring pixels; For pixels The neighborhood window (e.g., 5×5); For neighboring pixels The corrected passenger flow density at the location; An adaptive Gaussian kernel function; Indicates bandwidth; Represents the pixels to be smoothed With neighboring pixels Pixel coordinate interpolation in the horizontal direction; Represents the pixels to be smoothed With neighboring pixels Interpolation of pixel coordinates in the vertical direction.

[0235] Step S140: Input the multimodal fusion feature vector of historical time series into the pre-built time series prediction model to obtain the predicted total passenger flow and the predicted regional passenger flow density distribution for future times.

[0236] It should be understood that the specific model and its structure of the time series prediction model can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0237] Optionally, see [link to relevant documentation] Figure 2 The time-series prediction model includes a modality-aware temporal coding layer, a gated attention encoder layer, and a spatiotemporal dual-branch decoder layer;

[0238] This modality-aware temporal coding layer is used to inject temporal position coding and modal feature coding into each multimodal fusion feature vector of the historical time series; wherein, the modal feature coding is obtained by concatenating the scaling weight vectors corresponding to each data in the multimodal data and then performing a linear transformation;

[0239] This gated attention encoder layer is used to encode each multimodal fusion feature vector after injection temporal position encoding and modal feature encoding using a gated attention mechanism; wherein, in the gated attention mechanism, the gated vector generated by the quality factors corresponding to each data contained in the multimodal data is multiplied element-wise with the standard Transformer attention output;

[0240] This spatiotemporal dual-branch decoder layer is used to predict the total passenger flow and the regional passenger flow density distribution from the encoded features output by the gated attention encoder layer, and outputs the predicted total passenger flow and the predicted regional passenger flow density distribution.

[0241] Specifically, this solution innovatively proposes a time-series prediction model MTS-Former (Multi-modal Time-Series Transformer) optimized for subway scenarios. MTS-Former uses the multi-modal fusion feature vector F output in step S120 as the model input. Through three core innovative modules—modal-aware temporal coding, gated attention coding, and spatiotemporal dual-branch decoding—it achieves joint and accurate prediction of total passenger flow and regional heat map.

[0242] First, the time-series prediction model outputs the standardized multimodal fusion feature sequence in step S120. This serves as the model input. Specifically, this multimodal fusion feature sequence... It is constructed by sequentially arranging the multimodal fusion feature vectors corresponding to each time step in the historical time step in chronological order.

[0243] ;

[0244] ;

[0245] In the formula, T is the length of the input sequence (historical window), such as T=12, which corresponds to the past 60 minutes, 5 minutes / step; D represents the fusion feature dimension (D=128, consistent with step S120).

[0246] Modality-aware temporal coding layer:

[0247] Traditional Transformer architectures typically only use positional encoding (PE), failing to recognize the modal composition differences within the fused features. Therefore, this application designs a modality-aware temporal encoding that explicitly injects temporal positional information and modality reliability information into the features:

[0248] ;

[0249] In the formula, This represents the multimodal fusion feature vector after being enhanced by the modality-aware temporal coding layer at the t-th time step; This represents the multimodal fusion feature vector at time t. This represents the temporal position code at time t. This represents the modal feature encoding at time t.

[0250] Among them, the temporal position coding can adopt sine-cosine position coding to represent the absolute position of the feature in the temporal sequence.

[0251] ;

[0252] ;

[0253] In the formula, k is the feature dimension index ( ); t represents the time step index ( ).

[0254] Furthermore, the modal dynamic weights obtained from step S120 are obtained through linear transformation, allowing the model to perceive the data quality of each modality at the current moment:

[0255] ;

[0256] In the formula, This represents a linear layer that maps the dynamic weights of the three types of data to D-dimensional encoding, and... Dimension alignment.

[0257] Gated attention encoder layer:

[0258] The encoder is composed of N layers (e.g., N=3) of adaptive gated multi-head attention layers and feedforward network layers. It injects the dynamic modal weights of step S120 into the attention mechanism to achieve data quality-driven adaptive attention intensity.

[0259] First, this solution addresses the issue of standard Transformer attention. Based on the calculation, modal gating vectors are introduced. This allows for element-wise control of the intensity of attention output; the higher the modal weight, the stronger the attention in that dimension.

[0260] ;

[0261] ;

[0262] ;

[0263] In the formula, Indicates gating attention output; This represents the standard multi-head attention output; This represents element-wise multiplication; represents the modal gating vector at time step t; softmax represents the softmax function; K represents the query matrix; K represents the key matrix; V represents the value matrix; the superscript T represents the transpose matrix; d k The dimension representing each attention head; represents the Sigmoid activation function, which restricts the output to the (0,1) interval; Linear represents a linear transformation layer.

[0264] Secondly, the D-dimensional features are split into h heads (e.g., h=8), and each head independently computes adaptive gating attention. This multi-head mechanism allows the model to simultaneously focus on dependencies at different temporal scales.

[0265] ;

[0266] ;

[0267] ;

[0268] In the formula, It is a trainable parameter matrix; This represents the output of the i-th head; This represents the gated attention function; i represents the index of the attention head; This represents the final output of the multi-head attention mechanism.

[0269] Spatiotemporal dual-branch decoder layer:

[0270] High-dimensional temporal features output by the gated attention encoder layer R represents the real number field, and T represents the number of time steps. This high-dimensional temporal feature enters two independent decoding branches to complete the prediction of total passenger flow and the prediction of regional heat map, respectively, realizing one inference and spatiotemporal dual output.

[0271] One of the routes is a branch route with predicted total passenger flow:

[0272] This branch is used to predict the total number of station passengers in the next N steps. .

[0273] First, global features from the entire historical window are extracted using pooling operations to predict future total trends. This scheme performs global average pooling on the encoder output features to compress the temporal dimension.

[0274] ;

[0275] In the formula, This represents the global feature vector after pooling; This indicates the global average pooling layer.

[0276] Secondly, the pooled time-series information is fed into two fully connected layers, and decoded by the fully connected layers to obtain the total number of passengers for the next N steps:

[0277] ;

[0278] In the formula, This represents the total number of passengers in the next N steps; This indicates a fully connected layer, with a two-layer structure of D→64→N.

[0279] The other route is a regional thermal prediction branch:

[0280] This branch is used to predict the passenger flow density sequence of each area over the next N steps. , This indicates the number of grid cells in the station area, such as a platform level divided into 3×8.

[0281] First, the temporal features output by the encoder layer... Mapping to spatial features, upsampling is achieved using transposed convolution (Deconv). Spatial resolution is gradually restored through transposed convolution, decoding temporal features into spatial distributions:

[0282] ;

[0283] In the formula, The spatial feature tensor represents the result of upsampling the temporal features output by the encoder through transposed convolution; Deconv represents transposed convolution; T represents the number of time steps; and C represents the number of intermediate channels, i.e., the number of feature map channels output by the transposed convolution layer.

[0284] Secondly, a convolutional layer (conv) is introduced to extract spatial local dependencies from the upsampling results, and finally the results are mapped to passenger flow density:

[0285] ;

[0286] In the formula, This indicates passenger flow density.

[0287] Furthermore, during model training, both branch tasks are optimized simultaneously, and the total loss is calculated:

[0288] ;

[0289] ;

[0290] ;

[0291] In the formula, Weighting based on total passenger flow (e.g.) =0.4); Weighted by passenger flow density (e.g.) =0.6); The loss function for predicting total passenger flow for branch routes; N represents the number of prediction time steps; This represents the actual total passenger flow at the i-th time step in the future; This represents the predicted total passenger flow at the i-th time step in the future; The loss function for the regional thermal prediction branch; H and W represent the height and width of the station area grid, respectively; This represents the actual passenger flow density of the grid cell in the h-th row and w-th column at the i-th time step in the future; This represents the predicted passenger flow density of the grid cell in the h-th row and w-th column at the i-th time step in the future.

[0292] Step S150: Based on the pixel-level passenger flow density heat map, the predicted total passenger flow, and the predicted regional passenger flow density distribution, the station passenger flow control level is determined, and a control plan corresponding to the station passenger flow control level is generated.

[0293] It should be understood that the specific scheme for jointly determining the station's passenger flow control level based on the pixel-level passenger flow density heat map, the predicted total passenger flow, and the predicted regional passenger flow density distribution, and generating a control scheme corresponding to the station's passenger flow control level, can be set according to actual needs, and the embodiments of this application are not limited thereto.

[0294] Optionally, real-time passenger flow density is obtained from a pixel-level passenger flow density heatmap, and the passenger flow increase during the predicted period is calculated based on the predicted total passenger flow. The distribution of congested areas is determined based on the passenger flow density distribution in the predicted area. The real-time passenger flow density, passenger flow increase, and preset abnormal event trigger signals are compared with preset grading judgment conditions. Each grading judgment condition defines a corresponding density threshold condition, increase threshold condition, and abnormal event trigger condition. When at least two judgment conditions corresponding to the corresponding grading are met, the corresponding grading is determined as the station passenger flow control level. Based on the determined station passenger flow control level and congested area distribution, a pre-built grading control strategy knowledge base is retrieved to generate a control plan.

[0295] Specifically, based on the pixel-level passenger flow density heat map, the real-time passenger flow density values ​​of each area of ​​the station are extracted; based on the predicted total passenger flow, the percentage increase in passenger flow within a preset time period (such as 15 minutes) is calculated; and based on the predicted regional passenger flow density distribution, congested areas with passenger flow aggregation trends and their distribution range are identified.

[0296] See also Figure 2 The system compares real-time passenger flow density, passenger flow increase, and preset abnormal event trigger signals (such as the end of large-scale events or severe weather) against preset three-level passenger flow judgment conditions one by one. Specifically, the three-level passenger flow judgment conditions include: Level 1 judgment condition: real-time passenger flow density is less than 1.0 people / m². 2 The first-level judgment condition is that if the predicted increase in passenger flow within 15 minutes is less than 10% and no abnormal events are triggered, the passenger flow is stable and no control measures need to be initiated; the second-level judgment condition is that the real-time passenger flow density is between 1.0 and 1.5 people / m². 2 If the predicted passenger flow increase is between 10% and 30% within 15 minutes and there is a tendency for localized clustering, there is a risk of congestion, and preventative control measures need to be implemented; the third-level judgment condition is that the real-time passenger flow density reaches 1.5 people / m². 2If the passenger flow is predicted to increase by 30% or more within 15 minutes, or if congestion occurs simultaneously in multiple areas, or if events such as the end of a large-scale event or severe weather trigger a situation, a safety hazard exists, and emergency control measures must be initiated. When at least two of the following criteria for a given level are met: real-time passenger flow density, passenger flow increase, and preset abnormal event trigger signals, that level is determined as the station's passenger flow control level.

[0297] Subsequently, based on the determined station passenger flow control level and congestion area distribution, a pre-built hierarchical control strategy knowledge base is retrieved to generate a control plan adapted to the current passenger flow level and congestion area. This strategy knowledge base is pre-built based on subway operation specifications and historical control experience, covering differentiated control measures for different passenger flow levels and different congestion areas. Its core dimensions include: control objects (turnstiles, security checkpoints, escalators, entrances / exits, broadcast systems), control methods (flow restriction, diversion, guidance, and broadcast prompts), implementing entities (equipment systems, operations personnel, and dispatch center), and response time (Level 1 strategies are executed in real time, Level 2 strategies respond within 1 minute, and Level 3 strategies respond within 30 seconds). When generating control plans, the system prioritizes matching historical successful control cases with the same passenger flow level and similar congestion areas as the current situation, filters out unexecutable strategies based on real-time station equipment status, and dynamically adjusts the strategy intensity according to predicted passenger flow trends. For example, when passenger flow is predicted to continue to increase, the intensity level of control measures is automatically upgraded. The final control plan includes complete content such as control objectives, execution steps, responsible parties, response time and expected results, and supports output in three formats: text, voice and command stream, to adapt to the needs of different terminals.

[0298] After the control plan is generated, it is automatically executed by the station operation control platform through a standardized interface: For turnstiles and security checkpoints, the system automatically adjusts the turnstile passage mode (entry / exit / closed) and security checkpoint priority, and provides real-time feedback on equipment execution status; for the broadcast system, it automatically pushes multiple versions of broadcast prompts (congestion warning, diversion guidance, emergency control announcement) according to the plan content, supporting area-specific broadcasts; for escalators and entrances / exits, it remotely adjusts the escalator running direction and temporarily opens backup entrances / exits to achieve physical diversion of passenger flow. Simultaneously, the control tasks are pushed to on-site personnel through multi-terminal collaboration: the location of congested areas, control task details, and guidance scripts are pushed to on-site staff wearing devices, supporting voice broadcasting and one-click feedback; the station control room terminal pushes the plan execution progress, equipment status, and passenger flow trends, supporting visual monitoring and remote command. During the execution process, real-time data on changes in passenger flow, equipment status, and video surveillance are collected after the control measures are implemented. The predicted results are compared with the actual changes in passenger flow, and control effectiveness indicators such as congestion relief rate and average passenger flow speed improvement rate are calculated. The evaluation results, along with the corresponding scenarios and control plans, are fed back to the strategy knowledge base to update the optimal control strategies for different scenarios and achieve dynamic iterative optimization of the knowledge base.

[0299] In summary, by employing the above technical solutions, this application achieves the following technical effects:

[0300] First, this application uses an adaptive dynamic weight allocation mechanism based on data quality factors to independently evaluate the real-time quality of multimodal data such as video, gate passenger flow, and external correlations. It also dynamically adjusts the contribution weight of each modality during fusion based on the quality factors. When the quality of a certain modality deteriorates due to camera obstruction, equipment failure, or data delay, its fusion weight is automatically reduced, and other highly reliable modalities automatically fill the gap. This achieves intelligent complementarity and fault tolerance between modalities, significantly improving the robustness of multimodal fusion.

[0301] Secondly, in terms of passenger flow perception, this application uses a pixel-level heat map generation algorithm based on pedestrian kernel density estimation. It establishes a Gaussian distribution with pedestrian center coordinates and superimposes it pixel by pixel. Combined with the occlusion adaptive weight determined by the video quality factor, it performs density compensation correction on the occluded area. Then, through density adaptive Gaussian smooth interpolation, it realizes the generation of a pixel-level passenger flow density heat map under the panoramic view of the whole station. This makes the heat map accurately correspond to the physical scene and completely solves the problem of traditional fixed grid heat maps that "only show density and do not correspond to the scene".

[0302] Furthermore, in terms of short-term passenger flow prediction, this application constructs a temporal prediction model that includes a modality-aware coding layer, a gated attention encoder, and a spatiotemporal dual-branch decoder. The quality factor information generated during the multimodal fusion process is injected into each time step of the coding layer through modal feature encoding, and the gating vector generated by the quality factor in the attention calculation controls the attention output element by element. At the same time, the total passenger flow and regional passenger flow density distribution in the future period are predicted separately through two paths, realizing multi-dimensional joint accurate prediction.

[0303] Furthermore, in terms of generating control plans, this application constructs a hierarchical control strategy knowledge base based on subway operation standards and historical control experience. It determines the passenger flow control level by combining multiple conditions based on real-time passenger flow density, predicted passenger flow increase, and preset abnormal event trigger signals. It automatically generates control plans adapted to the current scenario by combining the distribution of congested areas, realizing a leap from fixed plans to dynamic intelligent generation, making the control plans more targeted and adaptable.

[0304] Furthermore, in terms of control and management implementation, this application establishes a closed-loop mechanism of "execution-feedback-optimization". The control and management plan is automatically executed by the station equipment through a standardized interface. At the same time, passenger flow change data after execution is collected, control and management effectiveness indicators are calculated and fed back to the strategy knowledge base, realizing dynamic iterative optimization of the knowledge base and enabling the system to have continuous learning and self-evolution capabilities.

[0305] Overall, this application constructs a complete closed loop from multimodal data acquisition, adaptive fusion, panoramic heat map generation, short-term passenger flow prediction to hierarchical contingency plan generation and coordinated execution, realizing full-link automation of passenger flow prediction and control in subway stations, significantly reducing reliance on manual intervention, and improving the real-time performance and accuracy of passenger flow control.

[0306] It should be understood that the above-mentioned subway passenger flow prediction and control method is merely exemplary, and those skilled in the art can make various modifications based on the above method, and the modified solutions also fall within the protection scope of this application.

[0307] This application provides a storage medium storing a computer program that, when executed by a processor, implements the steps of the subway passenger flow prediction and control method described above.

[0308] This application also provides an electronic device, including:

[0309] processor;

[0310] Memory is used to store computer programs that can be executed by a processor;

[0311] The steps involved in the processor executing the computer program include implementing the subway passenger flow prediction and control method described above.

[0312] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0313] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0314] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0315] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0316] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0317] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A metro passenger flow prediction and control method, characterized in that, include: Collect multimodal data from subway stations; wherein, the multimodal data includes passenger flow video data; The quality factor of each data point contained in the multimodal data is determined, and gating fusion is performed on all data points contained in the multimodal data based on each determined quality factor to generate a multimodal fusion feature vector; Based on the passenger flow video data, panoramic stitching is performed to obtain a full-station panoramic video image. Pedestrian kernel density estimation is used to map each pedestrian in the full-station panoramic video image to a Gaussian distribution, and all the Gaussian distributions are superimposed to obtain the superimposed result. The superimposed result is corrected by occlusion adaptive weight determined by the video quality factor corresponding to the passenger flow video data, and density adaptive Gaussian smooth interpolation is performed to generate a pixel-level passenger flow density heatmap. The multimodal fusion feature vectors of historical time series are input into a pre-built time series prediction model to obtain the predicted total passenger flow and the predicted regional passenger flow density distribution for future times. Based on the pixel-level passenger flow density heatmap, the predicted total passenger flow, and the predicted regional passenger flow density distribution, the station's passenger flow control level is determined, and a control plan corresponding to the station's passenger flow control level is generated.

2. The subway passenger flow prediction and control method according to claim 1, characterized in that, The multimodal data also includes gate passenger flow statistics and external data associated with passenger flow; The passenger flow data quality factor of the gate passenger flow statistics is determined based on the ratio of the number of effective sampling points to the theoretical number of sampling points and the proportion of outliers in the effective sampling data. The external data quality factor of the external data is determined based on the data update delay time of the external data; The video quality factor is determined based on the image occlusion rate and image blur of the passenger flow video data. 3.The metro passenger flow prediction and control method according to claim 1, characterized in that, The process involves gating and fusing all data within the multimodal data based on each determined quality factor to generate a multimodal fusion feature vector. Based on the proportion of each quality factor in the sum of all quality factors, the dynamic fusion weights corresponding to each data point included in the multimodal data are determined. Cross-modal attention is calculated on the feature vectors corresponding to each data point in the multimodal data to obtain the gated fusion weights corresponding to each data point in the multimodal data. The dynamic fusion weights corresponding to each data point in the multimodal data are multiplied by the gated fusion weights to obtain a scaling weight vector. The scaling weight vector is then multiplied element-wise by the feature vector corresponding to that data point to obtain the product results corresponding to each data point in the multimodal data. Finally, the product results corresponding to all data points in the multimodal data are summed to obtain the multimodal fusion feature vector.

4. The subway passenger flow prediction and control method according to claim 1, characterized in that, The passenger flow video data is collected via cameras; the process of performing panoramic stitching based on the passenger flow video data to obtain a panoramic video view of the entire station includes: Based on the field of view and layout parameters of each camera, a three-dimensional spatial constraint model is established to determine the effective shooting area of ​​each camera; Based on the overlap of the effective shooting areas, adjacent cameras are determined. SIFT feature points and ORB feature points are extracted from the video frames captured by the adjacent cameras respectively, and the two types of feature points are fused to obtain the feature point set corresponding to each video frame. Feature point matching is performed on the feature point sets of video frames captured by the adjacent cameras, and the homography matrix between adjacent video frames is solved based on the matching results. Based on the homography matrix, the video frames captured by each camera participating in the panoramic stitching are mapped, aligned and fused to obtain the full-site panoramic video image.

5. The subway passenger flow prediction and control method according to claim 4, characterized in that, The method involves using pedestrian kernel density estimation to map each pedestrian in the panoramic video footage of the entire site to a Gaussian distribution and then superimposing all the Gaussian distributions to obtain the superimposed result, including: Pedestrian target detection is performed on the entire panoramic video footage to obtain the center coordinates and size of each pedestrian; Using the center coordinates of each pedestrian as the center, each pedestrian is mapped to a Gaussian distribution, and the values ​​of the Gaussian distribution of all pedestrians at each pixel of the panoramic video of the whole site are weighted and summed with the size of each pedestrian as the weight to obtain the basic passenger flow density of each pixel; After traversing all pixels, an initial passenger flow density map is constructed from the basic passenger flow density of each pixel, and the initial passenger flow density map is used as the superposition result.

6. The subway passenger flow prediction and control method according to claim 5, characterized in that, The step of correcting the overlay result using an adaptive occlusion weight determined by the video quality factor corresponding to the passenger flow video data, and performing density adaptive Gaussian smooth interpolation to generate a pixel-level passenger flow density heatmap includes: The occlusion adaptive weight is determined based on the video quality factor; wherein the occlusion adaptive weight increases as the degree of occlusion represented by the video quality factor increases; The superposition result is corrected using the occlusion adaptive weight to obtain a corrected passenger flow density map; The modified passenger flow density map is subjected to density adaptive Gaussian smoothing interpolation to generate the pixel-level passenger flow density heatmap; wherein, the bandwidth of the Gaussian smoothing kernel used in the density adaptive Gaussian smoothing interpolation is dynamically adjusted according to the local passenger flow density at the smoothed pixel.

7. The subway passenger flow prediction and control method according to claim 3, characterized in that, The temporal prediction model includes a modality-aware temporal coding layer, a gated attention encoder layer, and a spatiotemporal dual-branch decoder layer; The modality-aware temporal coding layer is used to inject temporal position coding and modality feature coding into each of the multimodal fusion feature vectors in the historical time series; wherein, the modality feature coding is obtained by concatenating the scaling weight vectors corresponding to each data in the multimodal data and then performing a linear transformation; The gated attention encoder layer is used to encode each of the multimodal fusion feature vectors after injecting the temporal position encoding and the modal feature encoding using a gated attention mechanism; wherein, in the gated attention mechanism, the gated vector generated by the quality factor corresponding to each data contained in the multimodal data is multiplied element-wise with the standard Transformer attention output; The spatiotemporal dual-branch decoder layer is used to predict the total passenger flow and the regional passenger flow density distribution of the encoded features output by the gated attention encoder layer, and outputs the predicted total passenger flow and the predicted regional passenger flow density distribution. 8.The metro passenger flow prediction and control method according to claim 1, characterized in that, The step of jointly determining the station's passenger flow control level based on the pixel-level passenger flow density heatmap, the predicted total passenger flow, and the predicted regional passenger flow density distribution, and generating a control plan corresponding to the station's passenger flow control level, includes: The system obtains real-time passenger flow density based on the pixel-level passenger flow density heatmap, calculates the passenger flow increase during the predicted period based on the predicted total passenger flow, and determines the congestion area distribution based on the passenger flow density distribution in the predicted area. The real-time passenger flow density, the passenger flow increase, and the preset abnormal event trigger signal are compared with preset graded judgment conditions; wherein, each graded judgment condition defines a corresponding density threshold condition, increase threshold condition, and abnormal event trigger condition. When at least two of the judgment conditions corresponding to the corresponding level are met, the corresponding level is judged as the station passenger flow control level. Based on the determined station passenger flow control level and the distribution of congested areas, the pre-built hierarchical control strategy knowledge base is retrieved to generate the control plan.

9. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the subway passenger flow prediction and control method as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, it causes the at least one processor to perform the subway passenger flow prediction and control method as described in any one of claims 1 to 8.

Images