Urban public space disaster modeling and prevention system based on perception blind zone estimation

Abstract

The present invention provides a disaster modeling and prevention system for urban public spaces based on perception blind spot estimation, including six parts: local information representation, social force model simulator construction, local information fusion, global trajectory recovery, and multi-task joint abnormal event prediction. The present invention obtains local information representation within the sensor perception range based on multi-source heterogeneous data, obtains fragmentary trajectories and combines with social force model simulators, and proposes a trajectory fusion method suitable for multiple sensors in urban public spaces. Based on the two factors of direction and time, the fusion result is obtained by solving the optimal matching problem, and based on the social force model simulator, the complete perception in the urban public space is restored, solving the problem of difficulty in obtaining perception blind spot information; finally, based on the correlation between the occurrence relationships of multiple different abnormal events, joint training of multiple abnormal event detection or prediction tasks is carried out to achieve disaster modeling and prevention based on blind spot perception.

Description

Technical Field

[0001] The invention relates to a city public space disaster prevention system, belonging to the technical field of artificial intelligence. Background Art

[0002] With the development of modern society and the gradual improvement of economic and material conditions, sensing devices, such as temperature sensors, humidity sensors, CO2 sensors, surveillance cameras, etc., are widely used in the management of modern cities, especially public places, to improve the efficiency of urban operations, ensure the safety of public places, and enhance the happiness of citizens. Existing sensing technologies, such as crowd tracking, have been relatively mature in estimating the number of people within the sensing range of a single sensor.

[0003] However, in actual public places, the spatial coverage of a single sensor in public places has great limitations, which are mainly reflected in three aspects:

[0004] 1) The sensor itself has a limited sensing range. For example, a camera has a limited effective sensing area and is very restricted by the installation position and angle.

[0005] 2) The complexity of the public domain environment greatly limits the perception range of the sensor. The complexity of obstacles, stairs, elevators and other public facilities greatly affects the perception range and accuracy of cameras and sensors.

[0006] 3) The diversity of crowd behavior and the variability of crowd density in public areas will also affect the perception range of the sensor.

[0007] Therefore, it is difficult for a single sensor to achieve full coverage of the public environment. Even a perception network composed of multiple sensors will inevitably have perception blind spots. Existing perception technologies, whether based on a single sensor or a perception network composed of multiple sensors, are limited to the effective perception range of existing sensors and ignore perception blind spots. Taking crowd tracking as an example, single-camera crowd tracking technology focuses on crowd tracking based on video data extracted by a single camera, and multi-camera tracking considers the problem of trajectory fusion between cameras. However, existing methods mainly deal with outdoor scenes with sparse crowds. The trajectory fusion methods used are mainly greedy strategies, and less consideration is given to global optimization strategies. At the same time, these methods often only focus on pedestrian trajectories in the perceptible area, rather than pedestrian trajectories in the perception blind spots. However, in the public domain, these perception blind spots are also an important part of the public space, especially in places where sensors are sparsely distributed. Summary of the invention

[0008] Technical problem to be solved by the present invention: In order to solve the defects existing in the prior art, the present invention proposes an urban public space disaster modeling and prevention system based on perception blind spot estimation.

[0009] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0010] The present invention proposes an urban public space disaster modeling and prevention system based on perception blind area estimation, comprising the following steps:

[0011] Step S1, obtaining a local trajectory information representation within the sensing range of the sparse sensor, and obtaining a pedestrian trajectory within the sensing range of each sensor, that is, a segment trajectory set;

[0012] Step S2, draw a plan view of the environment, and calculate the point closest to each obstacle at each position based on the plan view information for query during the simulation process; construct a social force model simulator to calculate the social force on each pedestrian. When the social force model fails, perform path query and reselect intermediate targets by constructing a connected network diagram; output the state sequence of each pedestrian, and obtain the pedestrian trajectory during the simulation time period after processing; finally, adjust the parameters of the social force model simulator based on the local trajectory information of step S1;

[0013] Step S3, fragment trajectory fusion: through the transfer matrix, the fragment trajectory fusion problem is designed as a task allocation problem; then the Hungarian algorithm is used to solve the maximum matching optimization problem to obtain the fragment trajectory fusion sequence;

[0014] Step S4, restore the pedestrian trajectory according to the fragment trajectory fusion sequence obtained in step S3; use the social force model simulator to infer the blind spot trajectory, and generate the simulated pedestrian complete trajectory according to the output of the simulator;

[0015] Step S5: Use the crowd density detection model to detect the crowd density in the local area within a specific period of time, and correct the prediction result of the crowd movement trajectory segment in step S4; according to the obtained pedestrian trajectory, the pedestrian density in the perception blind area is counted to achieve modeling and prediction of various types of urban public space disasters.

[0016] Furthermore, in the urban public space disaster modeling and prevention system based on perception blind spot estimation proposed by the present invention, step S1 is specifically as follows:

[0017] Step 101: construct a training specific data set and manually annotate it; the data in the constructed data set includes: camera surveillance videos, gate card record, millimeter wave sensor data, and temperature and humidity sensor data in the subway station;

[0018] Step 102: Select a single sensor multi-target tracking framework, load pre-trained parameters, migrate to the labeled training data set, and perform training;

[0019] Step 103: Use the trained model to implement the trajectory recovery data set to obtain the pedestrian trajectory within the sensing range of each sensor, that is, the segment trajectory set.

[0020] Furthermore, in the urban public space disaster modeling and prevention system based on perception blind spot estimation proposed by the present invention, step S2 is specifically as follows:

[0021] Step 201: Draw an environment plan, rasterize the environment plan and store it as a dictionary, where the key is represented by an int type numeric symbol, K=(xx min )+(yy min )*h, the recorded value is 0, indicating that the location is passable, and -1, indicating that the location is occupied by an obstacle or boundary.

[0022] Step 202: Calculate the point closest to each obstacle at each position based on the geometric plane information, store it as a dictionary, and use it for query during simulation;

[0023] Step 203: construct a connected network graph, where the vertices of the network graph represent each grid at the location, and there is no inaccessible area on the grid. If two vertices are directly reachable, that is, one grid is in the eight adjacent directions of the other grid, then there is an edge between the two vertices. This connected graph is used for path query and intermediate target reselection when the social force model fails.

[0024] Step 204: At each moment, first add the pedestrians who have just entered the environment, and then determine whether there are any pedestrians who are stuck. If the movement range of a pedestrian in the first five steps is concentrated within the grid range, it means that the pedestrian cannot move normally based on the social force calculated based on the current target, and it is necessary to search for the shortest path and set a new target. This process is based on the established connectivity graph, and uses the Dijkstra shortest path algorithm to search for the shortest path with coarse precision, and then inserts the sampled position points on the shortest path into the new intermediate target.

[0025] Step 205: Calculate the social force on each pedestrian in the following way:

[0026]

[0027] m i is the mass of pedestrian i, is the acceleration of pedestrian i, They represent the three social forces on pedestrian i: driving force, obstacle force, and interaction force. The specific calculation is as follows:

[0028] 1) Driving force:

[0029]

[0030] where τ i is the relaxation time, which means the time required for the pedestrian to change speed to the ideal speed;

[0031] 2) Obstacle force:

[0032]

[0033] d io (t) represents the distance from pedestrian i to obstacle o at time t;

[0034] 3) Interactive force:

[0035]

[0036] r is the collision radius of the pedestrian.

[0037] The above calculations are used to obtain the acceleration of each pedestrian, and thus the travel speed at the current time step. Before updating the position, it is necessary to determine whether the pedestrian will enter an obstacle in the next step. If so, the travel speed is reduced according to the set attenuation. If this step is repeated three times and it is still impossible to move at the attenuated speed, the current speed is set to 0.

[0038] Step 206, repeating steps 204 and 205 until the time step is completed, and finally outputting the state sequence of each pedestrian, and obtaining the pedestrian trajectory in the simulation time period after processing;

[0039] Step 207: According to the local trajectory information obtained in step S1, adjust the parameters of the social force model simulator; the main parameters are set as follows: expected speed v0 = 1.5, A o =1.0, A i =2.7.

[0040] Furthermore, in the urban public space disaster modeling and prevention system based on perception blind spot estimation proposed by the present invention, step S3 is specifically as follows:

[0041] Step 301: By designing a transfer matrix, the segment trajectory fusion problem is designed as a task allocation problem, and its transfer matrix is ​​as follows:

[0042]

[0043] Where C1 represents the transfer matrix between two fragment trajectories,

[0044]

[0045] C2 represents the transition matrix of a segment trajectory as the trajectory termination segment,

[0046]

[0047] C3 represents the transition matrix of a certain segment trajectory as the initial segment of the trajectory,

[0048]

[0049] Step 302: Calculate the transfer matrix. For part C1, each element calculates the similarity between the two fragment trajectories. The similarity consists of two parts, one is based on time and the other is based on the moving direction. For parts C2 and C3, based on the prior information of the fragment trajectory, if the fragment trajectory k is the initial trajectory segment, then C (N+k)k = +∞, if the segment trajectory k is the initial trajectory segment, then C k(N+k) = +∞, N is the number of fragment trajectories;

[0050] Step 303: Using the Hungarian algorithm, solve the maximum matching optimization problem and obtain the trajectory segment fusion sequence result.

[0051] Furthermore, in the urban public space disaster modeling and prevention system based on perception blind spot estimation proposed by the present invention, step S4 is specifically as follows:

[0052] Step 401: construct the input file required for the social force model, including the pedestrian entry situation at each moment and the intermediate target point set of each pedestrian: obtain all local known segment trajectory sequences of each pedestrian according to the trajectory fusion result obtained in step S3; set some intermediate position points of these trajectory segment sequences as the intermediate targets of the pedestrians in the simulator, so as to guide the simulated intelligent agent to walk along the existing route as much as possible, thereby truly restoring the pedestrian trajectory;

[0053] Step 402: execute the social force model simulator, generate a simulated pedestrian trajectory according to the output of the simulator, and obtain a final restored complete trajectory.

[0054] Furthermore, the urban public space disaster modeling and prevention system based on perception blind zone estimation proposed in the present invention implements modeling and prediction of various urban public space disasters in step S5, specifically including:

[0055] Step 501: In view of the characteristics of dense smoke and open flames in fire, an image dataset containing dense smoke and open flames is collected, and the fire dataset is expanded using data augmentation methods including Rotation and RandomCrop. A deep convolutional network is used to construct a binary classification model of images containing open flames or smoke and not containing open flames or smoke, so as to model the fire and obtain a fire detection model, so as to achieve accurate recognition and alarm of images taken in fire scenes;

[0056] Step 502: The abnormal behaviors of pedestrians in the surveillance video are classified into two categories: abnormal objects and abnormal actions. Surveillance video data containing abnormal behaviors of pedestrians are collected, and an autoencoder model is trained with the goal of restoring the input video clips to model the abnormal behaviors of pedestrians in the surveillance video. When the autoencoder model cannot accurately restore the input video clips, it is considered that abnormal behaviors of pedestrians have occurred. For the features occurring at the fire scene in some scenes, the fire detection model is used to assist in marking the abnormality of the video clips, and the training architecture of distillation learning is used to guide the training of the abnormal behavior detection model of pedestrians, so as to achieve accurate time positioning and alarm of abnormal behavior events of pedestrians.

[0057] Step 503: in view of the correlation between abnormal crowd congestion and crowd density in urban public spaces, surveillance video data containing pedestrians is collected and a crowd density monitoring model in urban public spaces is constructed. When the crowd density exceeds a specific threshold, congestion is considered to have occurred, so as to model congestion and stampede disasters. In view of the fact that extreme changes in crowd density are often accompanied by abnormal pedestrian behavior, multi-task learning is used to jointly train a pedestrian abnormal behavior detection model and a crowd density detection model. In combination with the dilated convolution method, a crowd density estimation algorithm architecture of multi-scale feature extraction and fusion of an encoder-decoder structure is used, which is composed of a feature extraction module based on multi-column convolution, an encoder-decoder self-supervised training module, and a density regression module based on a multi-task architecture, so as to jointly realize accurate detection of abnormal crowd congestion in surveillance video images.

[0058] Step 504: using the crowd density detection model obtained in step 503 to detect the crowd density in the local area within a specific period of time, and correcting the prediction result of the crowd motion trajectory segment in step S4;

[0059] Step 505: The urban public space environment is counted as a 2m×2m grid. According to the pedestrian trajectory obtained in step S4, the pedestrian density in the environment, especially in the perception blind area, is counted to monitor congestion. For grids with a crowd density greater than δ, warning information is prompted, and intervention measures are taken in time to relieve congestion.

[0060] The present invention adopts the above technical solution, and has the following technical effects compared with the prior art:

[0061] (1) In view of the lack of research on blind spots in public places, this paper proposes a blind spot trajectory recovery scheme. By using a social force model simulator, the complete trajectory of pedestrians in the focused space can be restored through sparse perception data.

[0062] (2) For the single sensor tracking results, the present invention proposes a new fragment trajectory fusion scheme suitable for this scenario. It not only considers the time factor, but also makes use of spatial information as much as possible and takes the conversion cost of the moving direction into account. It can better fuse the fragment trajectories and restore the flow of people.

[0063] (3) Design a joint disaster prevention training program to predict pedestrian anomalies and anomalies in urban public spaces through restored trajectories so that preventive measures can be taken in a timely manner.

[0064] (4) Implement congestion prediction to predict the congestion that will occur in the environment in advance, so that managers can take appropriate measures.

[0065] (5) Implement “epidemic contact” query based on local perception for epidemic contact tracing in epidemic transmission management, thereby improving management efficiency and reducing management costs.

[0066] (6) Realize disaster modeling and visualization of urban public spaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is the architecture diagram of perception blind spot inference.

[0068] Figure 2 It is the disaster modeling task map.

[0069] Figure 3 This is the operation flow chart of the social force model.

[0070] Figure 4-7 The following are examples of trajectory recovery results, where (a) shows the true trajectory pairing, (b) shows the trajectory fusion result, and (c) shows the comparison between the recovered complete trajectory and the true trajectory.

[0071] Figure 8 This is an example of trajectory recovery failure, where (a) is the true trajectory pairing, (b) is the trajectory fusion result, and (c) is the comparison between the restored complete trajectory and the true trajectory.

[0072] Figure 9-12 is an example of congestion monitoring results, where (a) shows the actual crowd density and (b) shows the restored crowd density.

[0073] Figure 13-15 This is an example of congestion monitoring failure, where (a) shows the actual crowd density and (b) shows the restored crowd density.

[0074] Fig.16 This is a Truepositive example of epidemic contact query.

[0075] Fig.17 This is a false positive example of epidemic contact query.

[0076] Fig.18 This is a Falsenegative example of epidemic contact query.

[0077] Fig.19 This is an example of congestion prediction and prevention based on a crowd motion simulation model, where (a) shows the actual crowd density and (b) shows the predicted crowd density.

[0078] Fig. 20 Schematic diagram of the station emergency warning module in an embodiment of the present invention, wherein (a) is a schematic diagram of warning of possible crowd congestion events in the station in the future; (b) is a schematic diagram of issuing an alarm for open fire and thick smoke in the station. DETAILED DESCRIPTION

[0079] The technical solution of the present invention is now further described in conjunction with the accompanying drawings and embodiments. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0080] The present invention first obtains the local information representation of pedestrians within the perception range based on sparse sensors, extracts the set of pedestrian fragment trajectories, and combines the social force model simulator to fuse the fragment trajectories to obtain all the information of pedestrians in the known perception area, and designs a fragment trajectory fusion method based on the maximum matching problem. Subsequently, based on the fusion results of the fragment trajectories, the complete trajectory is estimated, especially the pedestrian trajectories in the blind area are obtained to obtain the crowd activity information of the entire area. Finally, various types of disaster events are modeled, and based on the correlation between the occurrence relationships of multiple different abnormal events, joint training of multiple abnormal event detection or prediction tasks is performed, and epidemic contact tracing is implemented to reduce management costs and improve management efficiency. At the same time, congestion prediction is implemented to predict the location of congestion in the subway station in advance, so that managers can take intervention measures immediately.

[0081] like Figure 1As shown, the present invention provides a technical solution for local perception that aggregates multi-source data, estimates the perception blind area, and obtains the global trajectory. Among them, local perception information such as gate card information, surveillance camera video, millimeter wave recording, temperature and humidity sensor recording, etc. are used for perception inference of blind areas, so as to obtain the complete trajectory of pedestrians, that is, global crowd perception. The obtained global crowd perception is used for multi-task joint training, and the tasks of congestion monitoring and prediction, abnormal behavior monitoring, smoke and fire disaster monitoring, etc., are used to train models with existing global information to complete the corresponding tasks.

[0082] like Figure 2 As shown, the present invention constructs a disaster modeling system in urban public spaces. Based on the collaborative interoperability mechanism and data aggregation and optimization, it integrates multi-source heterogeneous data to implement relevant detection and prediction tasks of disasters and anomalies in urban public spaces. Among them, the disaster modeling system in urban public spaces constructed by the present invention includes abnormal behavior detection of pedestrians, local congestion monitoring and prediction, fire and hazardous gas detection, and realizes comprehensive and systematic control of this series of anomalies, assisting regional managers to timely discover and take corresponding intervention measures.

[0083] The present invention models disasters in urban public spaces and constructs a corresponding detection system, and obtains local information representation based on sparsely deployed sensors, thereby inferring blind spots and realizing an urban public space disaster prevention system based on sparse sensor perception. The specific implementation steps of the present invention include:

[0084] Step 1: Use the following steps to obtain the local information representation within the sparse sensor perception range:

[0085] Step 1-1: Build a training-specific dataset, which is constructed from surveillance videos from cameras in subway stations, card-scanning records from gate machines, millimeter-wave sensor data, and temperature and humidity sensor data, and perform manual annotation;

[0086] Step 1-2: Select an existing single-sensor multi-target tracking framework, load the pre-trained parameters, migrate to the labeled training dataset, and perform training;

[0087] Step 1-3: Use the trained model to implement the trajectory recovery dataset to obtain the pedestrian trajectory in each camera, that is, the set of segment trajectories, where N is the number of segment trajectories.

[0088] Step 2: Follow the steps below to build a crowd simulator for the social force model. See the flowchart for the steps. Figure 3 :

[0089] Step 2-1: Draw the environment plan. To improve the running speed of the simulator, rasterize the environment plan into a rough map of 10 cm × 10 cm and store it as a dictionary. To save memory, the keywords are represented by int type digital symbols, K = (xx min )+(yy min )*h, the recorded value is 0, indicating that the location is passable, and -1, indicating that the location is occupied by an obstacle or boundary;

[0090] Step 2-2: Based on the geometric plan information, calculate the point closest to each obstacle at each position and store it as a dictionary for query during the simulation process. The calculation accuracy is still 10cm×10cm;

[0091] Step 2-3: Construct a connected network graph with an accuracy of 1m×1m. The vertices of the network graph represent each 1m×1m grid at the location, and there is no inaccessible area on the grid (to ensure the accessibility of the path). If two vertices are directly reachable, that is, one of the grids is in the eight adjacent directions of the other grid (east, southeast, south, southwest, west, northwest, north, northeast), then there is an edge between the two vertices. This connected graph is used for path query and intermediate target reselection when the social force model fails;

[0092] Step 2-4: At each moment, add new pedestrians into the environment, and then determine whether there are any stuck pedestrians. If the movement range of a pedestrian in the first five steps is concentrated within the range of 10cm×10cm, it means that the pedestrian cannot move normally based on the social force calculated based on the current target, and it is necessary to search for the shortest path and set a new target. This process is based on the established connectivity graph, and uses the Dijkstra shortest path algorithm to search for the shortest path with coarse precision, and then sample the position points on the shortest path and insert them into the new intermediate target;

[0093] Step 2-5: Calculate the social force on each pedestrian as follows:

[0094]

[0095] m i is the mass of pedestrian i, is the acceleration of pedestrian i, They represent the three social forces on pedestrian i, and their specific calculations are as follows:

[0096] 1) Driven force:

[0097]

[0098] where τ iis the relaxation time, which means the time required for the pedestrian to change its speed to the ideal speed. It can be understood that the movement of pedestrians is continuous and smooth. Usually, τ is set i =2.

[0099] 2) Obstacle force:

[0100]

[0101] d io (t) Record the distance from pedestrian i to obstacle o at time t.

[0102] 3)Interactive force:

[0103]

[0104] r is the collision radius of the pedestrian, which is usually 0.2m.

[0105] The above calculations are used to obtain the acceleration of each pedestrian, and thus the travel speed at the current time step. Before updating the position, it is necessary to determine whether the pedestrian will enter an obstacle in the next step. If so, the travel speed is reduced by a decay rate of 0.8. If this step is repeated three times and it is still impossible to move at the decayed speed, the current speed is set to 0.

[0106] Step 2-6: Repeat steps 2-4 and 2-5 until the time step is completed, and finally output the state sequence of each pedestrian, and obtain the pedestrian trajectory during the simulation time period after processing.

[0107] Step 2-7: According to the local trajectory information obtained in step 1, adjust the parameters of the social force model simulator. The main parameters are set as follows: expected speed v0 = 1.5, A o =1.0, A i =2.7.

[0108] Step 3: Follow the steps below to fuse the fragment trajectories into a fragment trajectory sequence. See the example of trajectory fusion results. Figure 4-Figure 8 Figures (a) and (b) are the actual track segment pairing and track segment fusion results, respectively. Figure 4 , Figure 5 , Figure 6 , Figure 7 is an example of trajectory recovery results, Figure 8 This is an example of trajectory recovery failure.

[0109] Step 3-1: By designing the transfer matrix, the fragment trajectory fusion problem is designed as a task allocation problem. The transfer matrix is ​​as follows:

[0110]

[0111] Where C1 represents the transfer matrix between two fragment trajectories,

[0112]

[0113] C2 represents the transition matrix of a segment trajectory as the trajectory termination segment,

[0114]

[0115] C3 represents the transition matrix of a certain segment trajectory as the initial segment of the trajectory,

[0116]

[0117] Step 3-2: Calculate the transfer matrix. For part C1, each element calculates the similarity between the two fragment trajectories. The similarity consists of two parts, one is based on time and the other is based on the moving direction. For parts C2 and C3, based on the prior information of the fragment trajectory, if the fragment trajectory k is the initial trajectory segment, then C (N+k)k = +∞, if the segment trajectory k is the initial trajectory segment, then C k(N+k) =+∞.

[0118] Step 3-3: Use the Hungarian algorithm to solve the maximum matching optimization problem and obtain the trajectory segment fusion sequence result.

[0119] Step 4: Use the social force model simulator to infer the blind area trajectory. See the example of trajectory recovery results. Figure 4-8 In Figure (c), the dotted line is the true trajectory, and the implementation is the restored trajectory.

[0120] Step 4-1: Construct the input file required for the social force model, including the pedestrian entry situation at each moment and the set of intermediate target points for each pedestrian: According to the trajectory fusion results obtained in step 3, all local known segment trajectory sequences of each pedestrian are obtained; some intermediate position points of these trajectory segment sequences are set as the intermediate targets of the pedestrians in the simulator, which are used to guide the simulated intelligent agent to walk along the existing route as much as possible, thereby truly restoring the pedestrian trajectory.

[0121] Step 4-2: Execute the social force model simulator, generate simulated pedestrian trajectories based on the output of the simulator, and obtain the final restored complete trajectory.

[0122] Step 5: Perform disaster modeling and prediction based on multi-task joint training according to the following steps:

[0123] Step 5-1: In view of the characteristics of thick smoke and open flames in fire, collect image datasets containing thick smoke and open flames. When thick smoke and open flames appear in the image, it is considered that a fire has occurred. Use data augmentation methods such as Rotation and RandomCrop to expand the fire dataset, and use a deep convolutional network to build a binary classification model for images with open flames or smoke and without open flames or smoke to model fires and achieve accurate recognition and alarm of images taken in fire scenes.

[0124] Step 5-2: The abnormal behaviors of pedestrians in the surveillance video are classified into two categories: abnormal objects and abnormal actions. The surveillance video data containing abnormal behaviors of pedestrians are collected, and the autoencoder model is trained with the goal of restoring the input video clips to model the abnormal behaviors of pedestrians in the surveillance video. When the autoencoder model cannot accurately restore the input video clips, it is considered that abnormal pedestrian behavior has occurred. In view of the characteristics of the fire scene in some scenes, the fire detection model is used to assist in marking the abnormality of the video clips, and the training architecture of distillation learning is used to guide the training of the abnormal pedestrian behavior detection model, so as to achieve accurate time positioning and alarm of abnormal pedestrian behavior events.

[0125] Step 5-3: In view of the correlation between abnormal crowd congestion and crowd density in urban public spaces, surveillance video data containing pedestrians is collected and a crowd density monitoring model for urban public spaces is constructed. When the crowd density exceeds a specific threshold, congestion is considered to have occurred, so as to model congestion and stampede disasters. In view of the fact that extreme changes in crowd density are often accompanied by abnormal pedestrian behavior, multi-task learning is used to jointly train the pedestrian abnormal behavior detection model and the crowd density detection model. Combined with the dilated convolution method, a crowd density estimation algorithm architecture with multi-scale feature extraction and fusion of an encoder-decoder structure is used, which consists of a feature extraction module based on multi-column convolution, a self-supervised training module based on an encoder-decoder, and a density regression module based on a multi-task architecture, which together achieve accurate detection of abnormal crowd congestion in surveillance video images and take timely intervention measures. Fig. 9 , Fig.10 , Fig.11 , Fig.12 is an example of congestion monitoring results, where (a) is the actual congestion situation and (b) is the restored congestion situation. Fig.13 , Fig.14 , Fig.15 This is an example of congestion monitoring failure, and the restored congestion situation is different from the actual situation.

[0126] Step 5-4: Use the crowd density detection model obtained in step 5-3 to detect the crowd density in the local area within a specific period of time, and correct the prediction result of the crowd motion trajectory segment in step 4.

[0127] Step 5-5: Count the environment in the urban public space into a 2m×2m grid. According to the pedestrian trajectory obtained in step 4, count the pedestrian density in the environment, especially in the blind spot range, to monitor congestion. For grids with a crowd density greater than δ, issue warning information and take timely intervention measures to relieve congestion.

[0128] Example 1: Crowd congestion prediction and prevention

[0129] With the development of society, large-scale activities are increasing, and the risk of crowded stampede accidents caused by large crowds gathering has also increased significantly. The characteristics of this type of disaster are that it occurs suddenly, has a major impact once it occurs, and is difficult to rescue. Regarding how to predict crowd congestion disasters and avoid stampede accidents, the present invention is based on the proposed disaster modeling and prediction technology based on multi-task joint training, which can assist management personnel to predict the time and place where congestion disasters may occur in advance, arrange personnel in advance to guide crowd movement, and reduce the risk of congestion stampede accidents caused by large crowds gathering.

[0130] like Figure 2 As shown in the figure, a crowd movement simulation model is constructed through the travel trajectory and map to predict the crowd density distribution in the future time period. The crowd density information of the local area is collected through monitoring cameras and millimeter wave sensors, and the crowd density distribution prediction results of the crowd movement simulation model are corrected in real time. When crowd congestion occurs in the map of the crowd movement simulation model, an alarm is issued to the management personnel. The management personnel can send staff to guide the crowd movement in advance to avoid crowd congestion and trampling risks.

[0131] Fig.19 This is an example of congestion prediction and prevention using a crowd movement simulation model. In the Unity map, the model predicts the time and location of crowd congestion, and then arranges robots to guide crowd movement in advance to avoid crowd congestion accidents in the future.

[0132] Therefore, by executing the method of the present invention, congestion prediction is implemented in the subway station, and the location where congestion is about to occur in the subway station is predicted in advance, which is convenient for management personnel to take intervention measures in time.

[0133] Example 2: Epidemic contact tracing and prevention and control

[0134] In the field of public health management, the tracking and tracing of the temporal and spatial correlation of epidemic virus carriers is one of the key points and difficulties in implementing epidemic management. In order to reduce the cost of management and control and improve the efficiency of epidemic contact screening, the present invention is based on the proposed sparse sensing-based disaster modeling and prevention technology, which can assist managers in accurately screening the temporal and spatial intersections of epidemic virus carriers, implement accurate screening, and improve management efficiency.

[0135] like Figure 1As shown in the figure, gates, temperature and humidity sensors, surveillance cameras, millimeter waves, etc. provide local perception of pedestrians in urban public spaces, combined with the inference of perception blind spots, and ultimately obtain the complete trajectory of pedestrians in the entire environment.

[0136] Based on the complete trajectory of pedestrians, "epidemic contacts" are defined and query operations are performed. "Epidemic contacts" refer to: within the same space, given a time range t and a spatial distance range s, for an abnormal pedestrian a, such as a confirmed carrier of an epidemic virus, other pedestrians on the path they pass through that meet the time interval t and spatial distance s are called a's epidemic contacts, that is, the key focus of epidemic prevention and control.

[0137] Fig.16 It is a Truepositive example of epidemic contact query, that is, the pedestrians who are actually real epidemic contacts among the epidemic contacts queried through trajectory recovery with reference to the groundtruth information.

[0138] Fig.17 This is a false positive example of epidemic contact query, that is, the pedestrians who are not actually epidemic contacts among the epidemic contacts queried through trajectory recovery with reference to the groundtruth information.

[0139] Fig.18 This is a False negative example of querying epidemic contacts, that is, referring to the groundtruth information, the pedestrian is actually a real epidemic contact, but is not found through trajectory recovery.

[0140] The present invention uses trajectory recovery for epidemic contact tracing and seeks out the spatiotemporal cross-population of epidemic spreaders, so as to achieve fine and accurate epidemic contact tracing, reduce control costs, and improve management efficiency.

[0141] Example 3: Urban public space disaster modeling and visualization

[0142] Driven by technologies such as AI, cloud computing, and big data analysis, the intelligent security industry has become increasingly open, and single video surveillance can no longer meet the needs of diversified security services. In order to achieve a monitoring and response mechanism for sudden abnormal conditions that perceives "precise, fast, wide, and detailed", this invention designs a robust detection and prediction model based on fusion data based on the characteristics of various abnormal events in subway stations, provides a visual interface to assist staff in decision-making, and reduces the harm caused by abnormal events.

[0143] like Fig. 20As shown in the figure, the emergency warning module in the station issues early warning prompts for abnormal events such as fire, harmful gas leakage, abnormal crowd behavior, local congestion, congestion prediction, etc. The emergency column will display the time of the emergency, the type of the abnormal event, and the type of sensor that identifies the abnormal event, so that the staff can handle the emergency in a timely manner.

[0144] Fig. 20 (a) warns of possible future crowd congestion in the station and lists the time and location of the possible congestion, as well as the inflow and outflow data of the station gates on which the congestion is based, to assist staff in determining the risk of crowd congestion in the station.

[0145] Fig. 20 (b) Alarms are sounded for open flames and thick smoke in the station, and the specific locations and detection times of the open flames and thick smoke, as well as the camera and gas sensor numbers based on them, are listed to assist staff in promptly discovering and handling fires in the station and reduce the losses caused by the fires.

[0146] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A system for modeling and preventing urban public space disasters based on perception blind zone estimation, characterized in that: The following steps are involved: Step S1, obtaining a local trajectory information representation within the sensing range of the sparse sensor, and obtaining a pedestrian trajectory within the sensing range of each sensor, that is, a segment trajectory set; Step S2, draw a plan view of the environment, and calculate the point closest to each obstacle at each position based on the plan view information for query during the simulation process; construct a social force model simulator to calculate the social force on each pedestrian. When the social force model fails, perform path query and reselect intermediate targets by constructing a connected network diagram; output the state sequence of each pedestrian, and obtain the pedestrian trajectory during the simulation time period after processing; finally, adjust the parameters of the social force model simulator based on the local trajectory information of step S1; Step S3, fragment trajectory fusion: through the transfer matrix, the fragment trajectory fusion problem is designed as a task allocation problem; then the Hungarian algorithm is used to solve the maximum matching optimization problem to obtain the fragment trajectory fusion sequence; Step S4, restore the pedestrian trajectory according to the fragment trajectory fusion sequence obtained in step S3; use the social force model simulator to infer the blind spot trajectory, and generate the simulated pedestrian complete trajectory according to the output of the simulator; Step S5: Use the crowd density detection model to detect the crowd density in the local area within a specific period of time, and correct the prediction result of the crowd movement trajectory segment in step S4; according to the obtained pedestrian trajectory, the pedestrian density in the perception blind area is counted to achieve modeling and prediction of various types of urban public space disasters.

2. The urban public space disaster modeling and prevention system based on perception blind area estimation according to claim 1 is characterized in that: Step S1 is specifically as follows: Step 101: construct a training specific data set and perform manual annotation; Step 102: Select a single sensor multi-target tracking framework, load pre-trained parameters, migrate to the labeled training data set, and perform training; Step 103: Use the trained model to implement the trajectory recovery data set to obtain the pedestrian trajectory within the sensing range of each sensor, that is, the segment trajectory set.

3. The urban public space disaster modeling and prevention system based on perception blind area estimation according to claim 1 is characterized in that: In step 101, the data in the constructed data set include: camera surveillance video, gate card record, millimeter wave sensor data, and temperature and humidity sensor data in the subway station.

4. The urban public space disaster modeling and prevention system based on perception blind area estimation according to claim 1 is characterized in that: Step S2 is specifically as follows: Step 201, draw an environment plan, rasterize the environment plan and store it as a dictionary; Step 202: Calculate the point closest to each obstacle at each position based on the geometric plane information, store it as a dictionary, and use it for query during simulation; Step 203: construct a connected network graph, where the vertices of the network graph represent each grid at the location, and there is no inaccessible area on the grid. If two vertices are directly reachable, that is, one grid is in the eight adjacent directions of the other grid, then there is an edge between the two vertices. This connected graph is used for path query and intermediate target reselection when the social force model fails. Step 204: At each moment, first add the pedestrians who have just entered the environment, and then determine whether there are any pedestrians who are stuck. If the movement range of a pedestrian in the first five steps is concentrated within the grid range, it means that the pedestrian cannot move normally based on the social force calculated based on the current target, and it is necessary to search for the shortest path and set a new target. This process is based on the established connectivity graph, and uses the Dijkstra shortest path algorithm to search for the shortest path with coarse precision, and then inserts the sampled position points on the shortest path into the new intermediate target. Step 205: Calculate the social force on each pedestrian in the following way: m i is the mass of pedestrian i, is the acceleration of pedestrian i, They represent the three social forces on pedestrian i: driving force, obstacle force, and interaction force; The above calculations are used to obtain the acceleration of each pedestrian, and thus the travel speed at the current time step. Before updating the position, it is necessary to determine whether the pedestrian will enter an obstacle in the next step. If so, the travel speed is reduced according to the set attenuation. If this step is repeated three times and it is still impossible to move at the attenuated speed, the current speed is set to 0. Step 206, repeating steps 204 and 205 until the time step is completed, and finally outputting the state sequence of each pedestrian, and obtaining the pedestrian trajectory in the simulation time period after processing; Step 207: Adjust the parameters of the social force model simulator according to the local trajectory information obtained in step S1.

5. The urban public space disaster modeling and prevention system based on perception blind area estimation according to claim 1 is characterized in that: The three social forces in step 205 are specifically calculated as follows: 1) Driving force: where τ i is the relaxation time, which means the time required for the pedestrian to change speed to the ideal speed; 2) Obstacle force: d io (t) represents the distance from pedestrian i to obstacle o at time t; 3) Interactive force: r is the collision radius of the pedestrian.

6. The urban public space disaster modeling and prevention system based on perception blind area estimation according to claim 1 is characterized in that: Step S3 is as follows: Step 301: By designing a transfer matrix, the segment trajectory fusion problem is designed as a task allocation problem, and its transfer matrix is ​​as follows: Where C1 represents the transfer matrix between two fragment trajectories, C2 represents the transition matrix of a segment trajectory as the trajectory termination segment, C3 represents the transfer matrix of a certain segment trajectory as the initial segment of the trajectory, Step 302: Calculate the transfer matrix. For part C1, each element calculates the similarity between the two fragment trajectories. The similarity consists of two parts, one is based on time and the other is based on the moving direction. For parts C2 and C3, based on the prior information of the fragment trajectory, if the fragment trajectory k is the initial trajectory segment, then C (N+k)k = +∞, if the segment trajectory k is the initial trajectory segment, then C k(N+k) = +∞, N is the number of fragment trajectories; Step 303: Using the Hungarian algorithm, solve the maximum matching optimization problem and obtain the trajectory segment fusion sequence result.

7. The urban public space disaster modeling and prevention system based on perception blind zone estimation according to claim 1 is characterized in that: Step S4 is specifically as follows: Step 401: construct the input file required for the social force model, including the pedestrian entry situation at each moment and the intermediate target point set of each pedestrian: obtain all local known segment trajectory sequences of each pedestrian according to the trajectory fusion result obtained in step S3; set some intermediate position points of these trajectory segment sequences as the intermediate targets of the pedestrians in the simulator, so as to guide the simulated intelligent agent to walk along the existing route as much as possible, thereby truly restoring the pedestrian trajectory; Step 402: execute the social force model simulator, generate a simulated pedestrian trajectory according to the output of the simulator, and obtain a final restored complete trajectory.

8. The urban public space disaster modeling and prevention system based on perception blind area estimation according to claim 1 is characterized in that: In step S5, various types of urban public space disaster modeling and prediction are implemented, including: Step 501: In view of the characteristics of dense smoke and open flames in fire, an image dataset containing dense smoke and open flames is collected, and the fire dataset is expanded using data augmentation methods including Rotation and RandomCrop. A deep convolutional network is used to construct a binary classification model of images containing open flames or smoke and not containing open flames or smoke, so as to model the fire and obtain a fire detection model, so as to achieve accurate recognition and alarm of images taken in fire scenes; Step 502: The abnormal behaviors of pedestrians in the surveillance video are classified into two categories: abnormal objects and abnormal actions. Surveillance video data containing abnormal behaviors of pedestrians are collected, and an autoencoder model is trained with the goal of restoring the input video clips to model the abnormal behaviors of pedestrians in the surveillance video. When the autoencoder model cannot accurately restore the input video clips, it is considered that abnormal behaviors of pedestrians have occurred. For the features occurring at the fire scene in some scenes, the fire detection model is used to assist in marking the abnormality of the video clips, and the training architecture of distillation learning is used to guide the training of the abnormal behavior detection model of pedestrians, so as to achieve accurate time positioning and alarm of abnormal behavior events of pedestrians. Step 503: in view of the correlation between abnormal crowd congestion and crowd density in urban public spaces, surveillance video data containing pedestrians is collected and a crowd density monitoring model in urban public spaces is constructed. When the crowd density exceeds a specific threshold, congestion is considered to have occurred, so as to model congestion and stampede disasters. In view of the fact that extreme changes in crowd density are often accompanied by abnormal pedestrian behavior, multi-task learning is used to jointly train a pedestrian abnormal behavior detection model and a crowd density detection model. In combination with the dilated convolution method, a crowd density estimation algorithm architecture of multi-scale feature extraction and fusion of an encoder-decoder structure is used, which is composed of a feature extraction module based on multi-column convolution, an encoder-decoder self-supervised training module, and a density regression module based on a multi-task architecture, so as to jointly realize accurate detection of abnormal crowd congestion in surveillance video images. Step 504: using the crowd density detection model obtained in step 503 to detect the crowd density in the local area within a specific period of time, and correcting the prediction result of the crowd motion trajectory segment in step S4; Step 505: The urban public space environment is counted as a 2m×2m grid. According to the pedestrian trajectory obtained in step S4, the pedestrian density in the environment, especially in the perception blind area, is counted to monitor congestion. For grids with a crowd density greater than δ, warning information is prompted, and intervention measures are taken in time to relieve congestion.

9. The urban public space disaster modeling and prevention system based on perception blind zone estimation according to claim 4 is characterized in that: In step 201, the environment plan is rasterized and stored as a dictionary, with the key represented by an int type numeric symbol, K = xx min +yy min *h, the recorded value is 0, indicating that the location is passable, and -1, indicating that the location is occupied by an obstacle or boundary.

10. The urban public space disaster modeling and prevention system based on perception blind zone estimation according to claim 4 is characterized in that: In step 207, the parameters of the social force model simulator are adjusted, and the parameters are set as follows: expected speed v0 = 1.5, A o =1.0, A i =2.7.

Images