Real-time data-driven crowd behavior modeling and simulation method for digital twinning

Abstract

The application discloses a kind of real-time data-driven crowd behavior modeling simulation methods for digital twinning, comprising: constructing three-dimensional twin static scene structure and two-dimensional simulation scene structure;Real-time acquisition of the dynamic changes of the crowd in dynamic scene structure in multiple time windows Video data;Obtain the macroscopic crowd movement flow and crowd density distribution of each time window;Macroscopic crowd movement flow and crowd density distribution are distributed to each unit area, obtain the movement flow set and cumulative density error of each unit area;Obtain the migration adjustment demand and migration adjustment amount of the agent in each unit area;The preferred speed of the agent in each unit area is obtained, the obstacle avoidance speed is obtained, and the three-dimensional dynamic crowd corresponding to the dynamic scene structure time is generated in three-dimensional twin static scene structure.The application can fully meet the application demand of digital twinning scene, with high fidelity, high restoration characteristics.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a real-time data driven crowd behavior modeling and simulation method for digital twinning. BACKGROUND

[0002] With the continuous development of the concept and technology of digital twinning, the digital twinning technology has important application value in building smart cities and further promoting the intelligentization and modernization of city governance. For the digital twinning scene, the crowd behavior modeling and simulation technology needs to efficiently process the dynamic data of the physical world crowd, so as to adjust the crowd behavior model in real time, and finally reproduce the real crowd dynamics in the virtual world. The real-time efficiency and fidelity of the data-driven crowd simulation as an important supporting technology of digital twinning directly affect the accuracy of the dynamic twinning crowd situation prediction. If the simulation model cannot be adaptively adjusted according to real-time information, when the twinning scene changes in crowd behavior mode over time, the simulated crowd will quickly deviate from the real behavior, thereby making it difficult to guarantee the accuracy and reliability of the crowd situation prediction.

[0003] The existing offline data-driven crowd simulation technology can only model and simulate the crowd behavior based on historical data, and cannot be adaptively adjusted according to real-time information, resulting in insufficient utilization of the rich crowd dynamic data provided by the digital twinning scene and inaccurate simulation prediction results. The existing online data-driven crowd simulation method needs to rely on high-quality micro pedestrian motion data (such as individual trajectory) to drive the simulation model; such micro data is difficult to efficiently and real-time extract in a large-scale crowd scene, and therefore is difficult to be effectively applied in the digital twinning scene. Therefore, how to utilize the crowd macro state real-time data which can be efficiently obtained to generate virtual world crowd behavior in real-time synchronization with the real scene, and realize efficient and accurate crowd situation prediction in the digital twinning scene is a technical problem to be solved by the present application. SUMMARY

[0004] In order to solve the above problems in the prior art, the present application provides a real-time data driven crowd behavior modeling and simulation method for digital twinning. The technical problem to be solved by the present application is solved by the following technical scheme:

[0005] In a first aspect, the present application provides a real-time data driven crowd behavior modeling and simulation method for digital twinning, comprising:

[0006] obtaining a scene structure; wherein the scene structure comprises a static scene structure and a dynamic scene structure;

[0007] constructing a three-dimensional twinned static scene structure and a two-dimensional simulation scene structure according to the static scene structure, and initializing the two-dimensional simulation scene structure;

[0008] Real-time acquisition of video data of dynamic changes of the crowd in the dynamic scene structure in multiple time windows;

[0009] According to the video data of each time window, the macroscopic crowd motion flow and the crowd density distribution of each time window are obtained;

[0010] The two-dimensional simulation scene structure is rasterized into a plurality of unit regions, the macroscopic crowd motion flow and the crowd density distribution are distributed into each unit region, and the motion flow set and the cumulative density error of each unit region are obtained.

[0011] The motion flow in the motion flow set of each unit region is matched with the current speed of the agent of each unit region, the direction of the motion flow in the motion flow set of each unit region is obtained, the migration adjustment demand and the migration adjustment amount of the agent in each unit region are obtained according to the cumulative density error of each unit region, and the acceleration of the agent in each unit region is obtained; wherein the agent represents a virtual pedestrian;

[0012] The preferred speed of the agent in each unit region is obtained according to the current speed and the acceleration of the agent in each unit region;

[0013] The collision avoidance speed is obtained according to the preferred speed of the agent in each unit region, and a three-dimensional dynamic crowd corresponding to the dynamic scene structure in time is generated in the three-dimensional twin static scene structure according to the collision avoidance speed.

[0014] The beneficial effects of the present application are:

[0015] 1. The real-time data-driven crowd behavior modeling simulation method for digital twinning provided by the present application can fully meet the application requirements of digital twinning scenes, and has the characteristics of high fidelity and high restoration.

[0016] 2. The real-time data-driven crowd behavior modeling simulation method for digital twinning provided by the present application extracts the macroscopic crowd state from the video, and compared with the method relying on micro individual trajectory information, the efficiency of data analysis and extraction is higher, and the performance will not suddenly decrease with the increase of the size of the crowd, so that the real-time requirement cannot be met.

[0017] 3. The real-time data-driven crowd behavior modeling simulation method for digital twinning provided by the present application uses real-time data to drive crowd simulation, can adaptively adjust the parameters of the agent according to the motion state and motion mode of the current crowd, and can accurately predict the situation in the scene with complex crowd structure and variable crowd motion mode such as large personnel flow, so as to improve the intelligent level of crowd safety management.

[0018] The present application will be further described in detail below in combination with the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0019] Fig. 1 is a flow chart of a real-time data driven crowd behavior modeling and simulation method for digital twinning provided by an embodiment of the present application;

[0020] Fig. 2 is a schematic diagram of a real-time data driven crowd behavior modeling and simulation device for digital twinning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] The present application will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0022] With the continuous advancement of smart city construction, advanced technologies represented by digital twinning have become a new path and practice mode for realizing digital and intelligent city safety management. According to the definition given by the United States National Aeronautics and Space Administration (NASA), digital twinning refers to making full use of physical models, sensors, operation history and other data, and integrating multidisciplinary, multi-physical quantity, multi-scale and multi-probability simulation processes. Unlike traditional modeling and simulation (M&S), digital twinning emphasizes "virtual-real integration", that is, physical space and digital space need to realize dynamic interaction. With the continuous development of the concept and technology of digital twinning, digital twinning technology has important application value in building smart cities and further promoting the intelligentization and modernization of city governance. At present, the application of digital twinning mainly reflects static visualization. Due to the low technical maturity in dynamic data analysis, simulation, and situation prediction, the depth of application scenarios is insufficient, and it is difficult to achieve "virtual control of reality".

[0023] Data-driven crowd simulation modeling technology is a key technology to realize the visualization and situation prediction of crowd dynamics in digital twinning. Data-driven crowd simulation modeling technology uses data captured in real environments to establish and improve crowd simulation models, which can effectively improve the realism of crowd simulation and has attracted widespread attention from domestic and foreign scholars. Current research in this field can be roughly divided into two categories: data-calibrated model-based methods and data-generated model-based methods. The data-calibrated model-based method mainly calibrates the important parameters in the given crowd simulation model, so that the simulation effect of the model approaches the real one. Zhong et al. used an evolutionary algorithm to calibrate the parameters of the social force model, so that the density distribution of the crowd in the simulation matched the video data. Luo et al. proposed a self-adaptive zooming evolutionary algorithm based on sampling and applied it to the parameter calibration of the social force model, which greatly improved the precision and computational efficiency of the model. Although this method can improve the realism of existing models to some extent, it cannot explain data that exceeds the representation ability of the model. The data-trained model-based method directly constructs a crowd simulation model using real crowd data, which can effectively extract the crowd motion rules in the data to make the simulation more realistic. Amirian et al. used a generative adversarial network to learn the trajectory data of real crowds and generate new trajectories with similar properties, which improved the realism of the simulation by embedding the generated trajectories into the crowd simulation. Zhao et al. extracted collision avoidance data of crowds from surveillance videos, summarized the patterns of pedestrian collision avoidance behavior through clustering algorithms, and trained a neural network model to convert them into behavior decisions in the simulation. Although existing technologies have made certain achievements in the establishment and optimization of data-driven crowd simulation models, when the simulation scene changes significantly in time and space, it will cause changes in crowd behavior patterns. The simulation model trained by existing technologies cannot adaptively adjust according to real-time information, and can only predict crowd behavior based on historical data, which will cause the simulation to quickly deviate from real behavior in new scenarios.

[0024] Beijing University of Technology proposed a data-driven crowd motion simulation method based on a generative adversarial network in its patent application "Data-driven crowd motion simulation method based on generative adversarial network", which generates virtual pedestrians that do not exist in the data set based on the pedestrian trajectory data set extracted from pedestrian motion video data in the simulation scene, and plans a complete path for the generated virtual pedestrians that is more realistic in response to given conditions such as initial position, destination, and other factors within the entire scene. This method applies a generative adversarial network based on a long short-term memory network to train the simulation model.

[0025] A crowd gathering and dispersing simulation system and method based on digital twin technology is disclosed in the patent application "Crowd gathering and dispersing simulation system and method based on digital twin technology" filed by Hebei Hanguang Heavy Industry Co., Ltd. The method includes: a user input module for user input of the simulation system's limiting conditions; a model establishment module for establishing a 1:1 digital three-dimensional model of a physical entity according to the user inputted limiting conditions; a physical entity data acquisition module for acquiring attribute data of the crowd speed and flow of the important monitoring points in the reality and feeding back the attribute data to the digital virtual body strategy generation module; a digital virtual body strategy generation module for generating and simulating the strategy of the gathering and dispersing process in the virtual simulation system; and a strategy iteration optimization module for fusing the physical entity data collected by the physical entity data acquisition module with the strategy data generated by the digital virtual body strategy generation module and optimizing the behavior and state of the physical entity in real time.

[0026] The defects of the related art are:

[0027] The crowd simulation based on offline data cannot adaptively adjust the model parameters according to the real-time crowd state, and the accuracy of the crowd behavior patterns that have not appeared in the simulation training scene is challenged.

[0028] The crowd simulation method based on online data needs to rely on a large amount of fine individual data, such as personnel age, gender ratio, individual motion trajectory, etc., which is difficult to obtain in real time in a large-scale crowd scene, limiting its application and promotion in complex scenes.

[0029] Therefore, the present application provides a real-time data-driven crowd behavior modeling simulation method for digital twin, which can efficiently extract the dynamic changes of the macro state of the crowd from real-time monitoring video data, solve the problems of relying on high-quality micro pedestrian data that is difficult to obtain in real time, the difficulty of adaptive adjustment of pedestrian motion patterns, and the low accuracy of long-term prediction results, and realize efficient simulation and situation prediction of crowd behavior in a dynamic, real-time and high-fidelity manner.

[0030] Please refer to Figs. 1-2 , Fig. 1 is a flowchart of the real-time data-driven crowd behavior modeling simulation method for digital twin provided by the embodiment of the present application, Fig. 2 is a schematic diagram of the real-time data-driven crowd behavior modeling simulation device for digital twin provided by the embodiment of the present application, and the real-time data-driven crowd behavior modeling simulation method for digital twin provided by the present application comprises:

[0031] S101, acquire the scene structure; wherein the scene structure includes static scene structure and dynamic scene structure.

[0032] Specifically, in this embodiment, according to the real world, the scene structure is acquired, including static scene structure and dynamic scene structure, the static scene structure includes obstacles, entrances and exits, scene shape and size, and the dynamic scene structure includes pedestrians, density distribution of pedestrians and pedestrian movement mode.

[0033] S102, according to the static scene structure, a three-dimensional twin static scene structure and a two-dimensional simulation scene structure are constructed, and the two-dimensional simulation scene structure is initialized.

[0034] Specifically, in this embodiment, the static scene structure information is collected and analyzed, a basic environment model with the same proportion as the real static scene structure is generated by using a three-dimensional modeling software Blender3D, the basic environment model is imported into a front-end 3D engine in a.glb format for rendering, and is converted into a static virtual environment, that is, a three-dimensional twin static scene structure, wherein the static virtual environment includes space size, obstacles, roads and environmental entrances and exits. At the same time, according to the static scene structure, a two-dimensional simulation scene structure is constructed, and according to the initial distribution of the crowd obtained from the real world, an agent is set in the simulation environment in a 1:1 manner to initialize the two-dimensional simulation scene structure, wherein the agent represents a pedestrian in the two-dimensional simulation scene structure.

[0035] S103, real-time video data of dynamic changes of the crowd in the dynamic scene structure in a plurality of time windows is collected.

[0036] Specifically, in this embodiment, a real-time streaming protocol (RTSP: Real Time Streaming Protocol) is used to establish a connection between the control end and the monitoring camera, the RTSP address of the camera is set through a link, and the real-time monitoring video stream is read and processed by the control end through the link.

[0037] A video stream data processing time window T is set, and a real-time monitoring video segment from the current time t0 to the time t0+T at the end of the time window is read. Since the monitoring video angle is often inclined, in order to more accurately capture the crowd movement state, a perspective transformation method is used to calibrate the inclined video angle to a top-down viewing angle. Each time window extracts a video segment with a length of T and provides it to a macro crowd movement state extraction module.

[0038] S104, according to the video data of each time window, a macro crowd movement flow and a crowd density distribution of each time window are acquired.

[0039] Specifically, in this embodiment, the macro crowd movement flow of each time window is acquired, including:

[0040] Sparse optical flow tracking is performed on the video data of each time window, and all sparse optical flows in each time window are acquired.

[0041] The sparse optical flow in each time window is clustered from three dimensions of time, space and optical flow direction to obtain clusters in each time window; when the clusters in each time window are greater than a threshold θ, the clusters in the time window form a macroscopic crowd motion flow, and the macroscopic crowd motion flow is represented by a binary tuple mf=(v, e), wherein v represents the direction of the macroscopic crowd motion flow, and e represents the flow of the direction of the macroscopic crowd motion flow.

[0042] In other words, sparse optical flow tracking is performed on the monitoring video segment. Specifically, feature points in the video that may represent pedestrians are identified through corner point detection, and under the premise of constant light and camera angle, the displacement of the corresponding feature points between video frames is the sparse optical flow. All sparse optical flows in a time window are tracked and recorded.

[0043] A grid-based density clustering method is used to cluster the sparse optical flow information in the time window from three dimensions of time, space and optical flow direction. When the number of sparse optical flows in the cluster exceeds the set threshold θ, it is considered that the optical flow cluster represents a macroscopic crowd motion flow, and is represented by a binary tuple mf=(v, e), wherein v represents the motion flow direction, which is obtained by averaging the directions of the optical flows in the cluster, and e represents the flow of the motion flow, which is represented by the number of optical flows in the cluster.

[0044] In this embodiment, the crowd density distribution of each time window is obtained, including:

[0045] The head features of pedestrians in the video data of each time window are extracted using the trained convolutional neural network, and the surrounding pixel coordinates of the head features of the pedestrians are assigned probabilities using a Gaussian kernel function to obtain the crowd density distribution of each time window.

[0046] In other words, the crowd density distribution map corresponding to the video frame at the beginning of the time window is extracted. First, the head features of the people in the video frame are extracted using a convolutional neural network, and the surrounding pixel coordinates are assigned probabilities based on the head features using a Gaussian kernel function, and a higher value indicates a higher probability of the presence of a human head at that location. All head features are processed to obtain a crowd distribution density map represented in the form of probabilities.

[0047] S105, the two-dimensional simulation scene structure is rasterized into a plurality of unit regions, the macroscopic crowd motion flow and the crowd density distribution are distributed to each unit region, and a motion flow set and a cumulative density error of each unit region are obtained.

[0048] Specifically, in this embodiment, the motion flow set of each unit region is obtained, including:

[0049] The positions of the sparse optical flows in the macroscopic crowd motion flow are distributed to the corresponding unit regions; for a unit region k, the motion flow set is represented as mf k,l L represents the total number of motion flows in the unit region k.

[0050] Specifically, the cumulative density error of each unit region is obtained, including:

[0051] The crowd density distribution is distributed into each unit region to obtain the position of the agent in the two-dimensional simulation scene structure;

[0052] According to the position of the agent in the two-dimensional simulation scene structure, the crowd distribution density in the two-dimensional simulation scene structure is calculated using a Gaussian kernel function; the simulation cumulative density of each unit region is obtained by integrating the crowd distribution density of each unit region

[0053] The cumulative density of each unit region is obtained by integrating the density of each pixel point in each unit region

[0054] According to the simulation cumulative density of each unit region and the cumulative density of each unit region The cumulative density error is obtained, that is

[0055] In other words, according to the size of the scene, the scene is divided into WxL grid unit regions, the macro crowd motion flow and the crowd density distribution are distributed into each unit region, and the unit region motion flow set and the unit region cumulative density are calculated.

[0056] The macro crowd motion flow is distributed into the corresponding unit region according to the position of its corresponding sparse optical flow distribution. For region k, the motion flow set is represented as where the motion flow is sorted in descending order of flow, and mf k,l L represents the total number of motion flows in the unit region k.

[0057] The unit region cumulative density is obtained by integrating the density of each pixel point in the unit region, and is represented as The crowd distribution density map in the simulation is calculated according to the position of the agent in the simulation using a Gaussian kernel function, and the simulation region cumulative density is obtained by integrating the density in the unit region The real world and simulation region cumulative density error of each unit region is calculated

[0058] S106. Match the motion flow in the motion flow set of each unit region with the current velocity of the agent in each unit region to obtain the direction of the motion flow in the motion flow set of each unit region; based on the cumulative density error of each unit region, obtain the migration adjustment requirements and migration adjustment amount of the agent in each unit region, and obtain the acceleration of the agent in each unit region; where the agent represents a virtual pedestrian.

[0059] Specifically, in this embodiment, the motion flow in the motion flow set of each unit region is matched with the current velocity of the agent in each unit region to obtain the direction of the motion flow in the motion flow set of each unit region, including:

[0060] Set the matching threshold for agents within each unit region to τ, and initialize it to 0.1; where agent a within unit region k... i The current speed is v i The matching threshold is τ i ;

[0061] Agent a within cell region k i Current speed v i The matching error is obtained by matching the motion flow with the motion flow set within the unit region k. Among them, matching error Indicates the current velocity v i With motion flow v k,l The included angle;

[0062] When matching error Less than the matching threshold τ i When that happens, the agent a within unit region k is determined. i Current speed v i Match the motion flow in the set of motion flows within the unit region k, and assign agent a i Current speed v i Update to match the motion flow direction; otherwise, agent a i Current speed v i If no match is found, update the matching threshold to τ. i +Δτ will reduce the matching error. Compare with the updated matching threshold until the matching error is reached. If the match threshold is less than the updated matching threshold, or if the updated matching threshold is greater than the preset upper bound, proxy a will be used. i Current speed v i Update to match the motion flow direction.

[0063] In this embodiment, obtaining the migration and adjustment requirements of agents within each unit area includes:

[0064] When the cumulative density error of each cell region is greater than 0, it indicates that the agent in the current cell region needs to be migrated and adjusted to other cell regions.

[0065] When the cumulative density error of each cell region is less than 0, it indicates that the agents in other cell regions need to be migrated and adjusted to the current cell region.

[0066] In this embodiment, obtaining the migration adjustment amount of the agent within each unit area includes:

[0067] The migration regulation of the agent in each cell region is estimated by minimizing the cumulative density error of each cell region; where the migration regulation is represented by an n×n matrix M, n represents the total number of cell regions, and m is the number of elements in matrix M. ij This represents the number of agents that need to be migrated from cell region i to cell region j. Matrix M is obtained by solving an optimization problem, where the optimization problem expression is:

[0068]

[0069] Among them, c ij b represents the cost of migrating the adjustment agent from cell region i to cell region j. i L represents the migration and adjustment requirements of unit region i. ij and U ij Let x and y represent the lower and upper bounds of the number of migration adjustment agents, respectively. The solution obtained by solving the optimization problem is the migration adjustment matrix M, where M represents the adjustment scheme at the unit region level, units represents the unit region, and x represents the adjustment scheme at the unit region level. ij The number of agents involved in the migration adjustment from cell region i to cell region j is represented by w, which represents the migration adjustment demand weight. ki This represents the proxy quantity for the migration adjustment from cell region k to cell region i.

[0070] In other words, within each cell region, the motion flow within the region is matched based on the agent's current velocity, and the velocity features are updated based on the matching results. The motion flow matching process for agents within a cell region is as follows: each agent is assigned a matching threshold τ, initialized to 0.1. Given agent a within cell region k... i Its current speed is v i The matching threshold is τ i a i sequentially with MF k Matching is performed on each motion stream in the process, and the matching error is... Defined as velocity v i and the direction of motion flow v k,l The angle between them. When the matching error is less than the matching threshold τ i When a match is confirmed, matching of other motion streams is stopped. iVelocity feature v i is updated to match the motion flow direction. When all motion flows in the region are matched, a i is not determined, τ i = τ i + Δτ is increased to increase the matching threshold. The motion flow matching process is performed again with the new matching threshold until a match is determined or the matching threshold exceeds the upper bound, and the matching is stopped.

[0071] According to the region cumulative density error Δρ k , the agent adjustment demand in the region is estimated, which is greater than 0 when the current number of simulation agents in the region is too large and needs to be adjusted to other regions; otherwise, it indicates that agents need to be adjusted from other unit regions to the current unit region. The migration adjustment amount between regions is estimated by minimizing the density error. The migration adjustment amount is defined as an n×n matrix M, where n is the total number of scene grids, and the elements m ij in the matrix are integers representing the number of agents that need to be adjusted from region i to region j. The migration adjustment amount matrix is calculated by solving the following optimization problem:

[0072]

[0073] where c ij represents the cost of adjusting agents from region i to region j, which is calculated according to the distance between regions and the similarity of motion flows between regions; b i represents the adjustment demand of region i, which is greater than 0 when agents need to be adjusted out and less than 0 when agents need to be adjusted in, and the adjustment demand is estimated according to the density error; L ij and U ij represent the lower and upper bounds of the number of agents to be migrated and adjusted, respectively.

[0074] Based on the migration adjustment amount, a stable matching algorithm is used to select agents from the region to perform adjustment. Specifically, according to the number of agents that need to be adjusted out, the largest partial agent with matching error is selected from the region. The acceptance function f1(a,u) of the agent to the region and the acceptance function f2(a,u) of the region to the agent are designed respectively. The two acceptance functions f1 and f2 between the agent to be adjusted and the region to be adjusted in are calculated and sorted respectively, and a stable matching problem between the agent and the region is constructed to find the best match, and the specific adjustment scheme at the individual level is obtained, i.e. whether each agent in each unit region needs to be migrated and adjusted, the speed of migration and adjustment, and the migration and adjustment to the corresponding other unit region.

[0075] According to the best match, the acceleration of the agent in each unit region is obtained.

[0076] Assuming that the agent a i in region k needs to be adjusted to region k', the acceleration αi The direction is the current position p of the agent i (x, y) to the center of the region k' The direction of the line, the size is calculated as Where T represents the size of the time window.

[0077] S107, according to the current speed and acceleration of the agent in each unit area, the preferred speed of the agent in each unit area is obtained.

[0078] Specifically, in this embodiment, based on the speed v i and acceleration a i learned by the hierarchical mapping module, the preferred speed of the agent is calculated in combination with the current simulation time t. The preferred speed is calculated as Update the preferred speed of all agents.

[0079] S108, according to the preferred speed of the agent in each unit area, the collision avoidance speed is obtained; according to the collision avoidance speed, a three-dimensional dynamic crowd corresponding to the time of the dynamic scene structure in the three-dimensional twin static scene structure is generated.

[0080] Specifically, in this embodiment, the obstacle avoidance speed is obtained, including:

[0081] The collision avoidance speed of the agent under the preferred speed is calculated using a collision avoidance algorithm, and the state of all agents in the simulation is updated with the collision avoidance speed. Repeat the above process at each simulation step within the time window T.

[0082] For the real-time video stream data obtained above, S104 is executed every time window to obtain a new macroscopic crowd motion flow and crowd density distribution map, and steps S105 to S106 are executed to drive the simulation model to adaptively adjust the parameters of the agent according to real-time information. The speed and position distribution of the agent in the simulation model are queried in real time, and a 3D dynamic crowd is generated in the 3D digital twin static scene at a time synchronized with the physical world.

[0083] When it is necessary to predict the crowd situation, a simulation copy is generated in parallel and is pushed forward to obtain the future speed and position information of the predicted agent from the simulation, and the distribution of the agent is mapped to the 3D digital twin scene to observe the changes of the crowd situation.

[0084] In summary, the beneficial effects of the present application are:

[0085] 1. The real-time data driven crowd behavior modeling and simulation method for digital twin provided by the present application can fully meet the application requirements of digital twin scene, and has the characteristics of high fidelity and high restoration.

[0086] 2. The real-time data-driven crowd behavior modeling simulation method for digital twinning provided by the application extracts macroscopic crowd states from videos, and compared with methods relying on micro individual trajectory information, the efficiency of data analysis and extraction is higher, and the performance will not suddenly decrease with the increase of the size of the crowd, thus meeting the real-time requirement.

[0087] 3. The real-time data-driven crowd behavior modeling simulation method for digital twinning provided by the application adopts real-time data to drive crowd simulation, can adaptively adjust the parameters of agents according to the motion state and motion mode of the current crowd, and can accurately predict the situation in scenes with large personnel flow, complex crowd structure and variable crowd motion mode, thus improving the intelligent level of crowd safety management.

[0088] Based on the same inventive concept, please continue to see Fig. 2 The application further provides a real-time data-driven crowd behavior modeling simulation device for digital twinning, which is applied to the real-time data-driven crowd behavior modeling simulation method for digital twinning provided by the above-mentioned embodiments of the application, and the method embodiments are described above and will not be repeated here, and the device comprises:

[0089] A digital twinning static scene generation module is configured to generate a virtual simulation environment consistent with the structure of a real scene.

[0090] A real-time data acquisition module is configured to acquire monitoring videos of main crowd motion scenes in a real world in real time and feed them back to a macroscopic crowd dynamic extraction module.

[0091] The macroscopic crowd dynamic extraction module is configured to identify macroscopic crowd dynamics such as crowd motion flow and density distribution in the videos.

[0092] A macro-to-micro hierarchical mapping module is configured to learn attribute features such as velocity and acceleration of physical entities from the macroscopic crowd dynamics and feed them back to a micro pedestrian motion simulation module.

[0093] The micro pedestrian motion simulation module is configured to adjust the motion of simulation agents in real time according to the attribute features.

[0094] A digital twinning crowd behavior generation module is configured to generate crowd behavior and situation prediction in real time in synchronization with a real scene.

[0095] Specifically, in this embodiment, the digital twin static scene generation module is configured to generate a virtual simulation environment consistent with the real scene structure; the real-time data acquisition module is configured to acquire monitoring videos of a main crowd movement scene in a real world in real time and feed back to the macro crowd dynamic extraction module; the macro crowd dynamic extraction module is configured to identify macro crowd dynamics such as crowd movement flow and density distribution in the videos and provide the macro-to-micro hierarchical mapping module; the macro-to-micro hierarchical mapping module is configured to learn attribute features such as speed and acceleration of physical entities from the macro crowd dynamics and feed back the attribute features to the micro pedestrian movement pattern establishment module; the micro pedestrian movement simulation module is configured to adjust the movement pattern of the simulation agent in real time according to the attribute features; and the digital twin crowd behavior generation module is configured to generate crowd behavior and situation prediction in real time in synchronization with the real scene.

[0096] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variant are intended to cover non-exclusive inclusion, so that the article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed. Without more limitation, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the article or device including the element. The terms "connected" or "connected" and the like are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", and the like are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0097] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present specification.

[0098] The above is further detailed description of the present application in combination with specific preferred embodiments, and cannot be deemed as limitation of the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, and all should be deemed as falling within the protection scope of the present application.

Claims

1. A real-time data-driven crowd behavior modeling and simulation method for digital twinning, characterized in that, The method comprises the following steps: acquiring a scene structure; wherein the scene structure comprises a static scene structure and a dynamic scene structure; constructing a three-dimensional twin static scene structure and a two-dimensional simulation scene structure according to the static scene structure, and initializing the two-dimensional simulation scene structure; collecting video data of dynamic changes of a crowd in the dynamic scene structure in a plurality of time windows in real time; acquiring a macroscopic crowd motion flow and a crowd density distribution of each of the time windows according to the video data of each of the time windows; gridizing the two-dimensional simulation scene structure into a plurality of unit regions, distributing the macroscopic crowd motion flow and the crowd density distribution into each of the unit regions, acquiring a motion flow set and a cumulative density error of each of the unit regions; matching a motion flow in the motion flow set of each of the unit regions with a current speed of an agent in each of the unit regions to acquire a direction of the motion flow in the motion flow set of each of the unit regions; acquiring a migration adjustment demand and a migration adjustment amount of the agent in each of the unit regions according to the cumulative density error of each of the unit regions, and acquiring an acceleration of the agent in each of the unit regions; wherein the agent represents a virtual pedestrian; wherein the acquisition of the migration adjustment demand of the agent in each of the unit regions comprises: when the cumulative density error of each of the unit regions is greater than 0, it indicates that the agent in the current unit region needs to be migrated and adjusted to other unit regions; when the cumulative density error of each of the unit regions is less than 0, it indicates that the agent in other unit regions needs to be migrated and adjusted to the current unit region; the acquisition of the migration adjustment amount of the agent in each of the unit regions comprises: The migration adjustment amount of the agent of each of the unit areas is estimated by minimizing the accumulated density error of each of the unit areas; wherein the migration adjustment amount is expressed as the matrix , represents the total number of unit areas, the elements in the matrix , represent the number of agents that need to be migrated from the unit area to the unit area , and the matrix is obtained by solving an optimization problem, wherein the optimization problem is expressed as: ； wherein, represents the number of migration regulating agents from the cell region to the cell region migration regulating agents, represents the cell region migration regulating demand, and respectively represent the lower bound and the upper bound of the number of migration regulating agents, represents the cell region, represents the number of migration regulating agents from the cell region to the cell region migration regulating agents, represents the migration regulating demand weight, represents the number of migration regulating agents from the cell region to the cell region migration regulating agents; the acquisition of the acceleration of the agent in each of the unit regions comprises: According to the migration adjustment amount of the agent in each unit area, select the agent with the largest matching error from each unit area, and set the acceptance function of the agent to the unit area respectively and the acceptance function of the unit area to the agent ; solving and , a stable matching problem between the agents and the cell regions is constructed and solved, and the optimal matching is obtained, that is, whether each agent in each of the cell regions needs to migrate and adjust, the speed of migration and adjustment, and migration and adjustment to the corresponding other cell region are obtained; acquiring the acceleration of the agent in each of the unit regions according to the best match; acquiring a preferred speed of the agent in each of the unit regions according to the current speed and the acceleration of the agent in each of the unit regions; acquiring an avoidance speed according to the preferred speed of the agent in each of the unit regions; and generating a three-dimensional dynamic crowd corresponding to the dynamic scene structure in time in the three-dimensional twin static scene structure according to the avoidance speed.

2. The digital twin oriented real-time data driven crowd behavior modeling and simulation method according to claim 1, characterized in that, the acquisition of the macroscopic crowd motion flow of each of the time windows comprises: performing sparse optical flow tracking on the video data of each of the time windows to acquire all sparse optical flows in each of the time windows; The sparse optical flow in each time window is clustered from three dimensions of time, space and optical flow direction to obtain clusters in each time window When the clusters in each time window are greater than a threshold value The clusters in the time window form a macroscopic crowd motion flow, and the macroscopic crowd motion flow is represented by a binary tuple Wherein, represents the direction of the macroscopic crowd motion flow, And represents the flow of the direction of the macroscopic crowd motion flow.

3. The digital twin oriented real-time data driven crowd behavior modeling and simulation method of claim 1, wherein, the acquisition of the crowd density distribution of each of the time windows comprises: extracting a head feature of a pedestrian in the video data of each of the time windows using a trained convolutional neural network, and assigning a probability to the surrounding pixel coordinates of the head feature of the pedestrian using a Gaussian kernel function to acquire the crowd density distribution of each of the time windows.

4. The digital twin oriented real-time data driven crowd behavior modeling and simulation method of claim 1, wherein, the acquisition of the motion flow set of each of the unit regions comprises: assigning locations of sparse optical flow distribution in the macroscopic crowd motion flow to corresponding individual cell regions; for a cell region , the motion flow set is represented as , represents the number of sub-motion flows within a cell region represents the number of sub-motion flows within a cell region represents the total number of motion flows within a cell region.​ 5. The digital twin oriented real-time data driven crowd behavior modeling and simulation method of claim 1, wherein, the acquisition of the cumulative density error of each of the unit regions comprises: distributing the crowd density distribution into each of the unit regions to obtain the position of the agent in the two-dimensional simulation scene structure; Based on the positions of the agents in the two-dimensional simulation scene structure, the crowd distribution density in the two-dimensional simulation scene structure is calculated using a Gaussian kernel function; the cumulative simulation density of each unit region is obtained by integrating the crowd distribution density of each unit region. ; The cumulative density of each of the unit regions is obtained by integrating the density of each pixel point in each of the unit regions ; According to the respective unit area simulation cumulative density and the cumulative density of each unit area , the cumulative density error is obtained, that is .

6. The digital twin oriented real-time data driven crowd behavior modeling and simulation method of claim 1, wherein, The matching of the motion flow in the motion flow set of each unit region with the current speed of the agent of each unit region obtains the direction of the motion flow in the motion flow set of each unit region, and includes: setting a matching threshold value of the agent in each of the unit areas as and initializing to 0.1; wherein the current speed of the agent in the unit area is , and the matching threshold value is . ; The unit region Internal Agent Current speed With the unit region Matching motion flows within the set of motion flows within the set yields the matching error. Wherein, the matching error Indicates the current speed With the flow of motion The included angle; When the matching error Less than the matching threshold When, the unit region is determined. Internal Agent Current speed With the unit region Match motion flows within the set of motion flows and assign them to agents. Current speed Update to match the motion flow direction; otherwise, agent Current speed If no match is found, update the matching threshold to... The matching error Compare with the updated matching threshold until the matching error is reached. If the match threshold is less than the updated matching threshold, or if the updated matching threshold is greater than the preset upper bound, the proxy will be used. Current speed Update to match the motion flow direction.

7. The digital twin oriented real-time data driven crowd behavior modeling and simulation method of claim 1, wherein, The obtaining of the collision avoidance speed includes: The collision avoidance speed of the agent under the preferred speed is calculated by using a collision avoidance algorithm.

Images