Real-time monitoring and calculation method of pedestrian flow in urban areas

Abstract

The present invention discloses a method for real-time monitoring and calculation of urban regional human flow, which divides the acquired boundary information of any urban area into production end, consumption end and task supplementary recording, wherein: the production end mainly realizes the real-time access function of spatiotemporal information data, completes data cleaning, judges data quality, generates schedules for high-quality data in real time, extracts features for low-quality data to generate intermediate tables, and stores them in data cache pool; the consumption end reads cache pool data, judges data quality, generates schedules for high-quality data in real time, and performs task scheduling according to the task status, execution order and restart times of each task marked by the schedule, and the task scheduling mainly includes calculation tasks and prediction tasks; task supplementary recording reads offline data, judges data quality, and calculates regional human flow. It solves the problem of real-time calculation of tens of millions of moving objects based on spatiotemporal big data, has the advantages of highly flexible streaming time window, high throughput, etc., the calculation results are more accurate, and offline supplementary recording results are supported.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and relates to the calculation of monitoring the flow of people in urban areas by using big data technology, and in particular to a real-time monitoring and calculation method of the flow of people in urban areas. Background Art

[0002] Big data technology has brought about a fundamental change in the way data and information are processed. Urban management should move towards a "smart governance" model, which emphasizes the extensive application of knowledge and technology represented by big data, provides data support for the country and the government to deal with urban management affairs, and improves the ability to predict in advance, control during the process, and monitor after the event. In public places with high population density in cities, such as scenic spots, shopping malls, stadiums, conference halls, etc., it is necessary to grasp the regional flow of people in real time, predict future flow trends, and provide timely and effective support for the relevant work of government departments.

[0003] At present, the real-time calculation method of big data mainly emphasizes the timeliness of consumption. However, from the business perspective, it is more important to consider the accuracy of the calculation results and how to more realistically reflect the actual situation. Since the business that requires real-time or quasi-real-time push, such as management and control, depends on the quality of data push, and the data itself is not completely continuous and the delay of front-end processing, it is necessary to design methods to compensate for the inaccurate business service results caused by incomplete data. Summary of the invention

[0004] In view of the defects or shortcomings in the prior art, the object of the present invention is to provide a real-time monitoring and calculation method for urban area pedestrian flow.

[0005] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0006] A real-time monitoring and calculation method for urban area pedestrian flow, characterized in that the method divides the acquired boundary information of any urban area into production end, consumption end, and task supplementary recording, wherein:

[0007] The production side mainly implements the real-time access function of spatiotemporal information data, completes data cleaning, judges data quality, generates real-time schedules for high-quality data, extracts features for low-quality data, generates intermediate tables, and stores them in the data cache pool;

[0008] On the consumer side, read the cache pool data, judge the data quality, generate a schedule for high-quality data in real time, and schedule tasks based on the task status, execution order, and restart times marked in the schedule. Task scheduling mainly includes computing tasks and prediction tasks.

[0009] Re-record tasks, read offline data, judge data quality, and calculate regional passenger flow.

[0010] According to the present invention, the calculation of regional crowd flow is divided into regional crowd flow real-time calculation and regional crowd flow prediction; wherein:

[0011] Real-time calculation of regional population flow: mainly refers to the real-time calculation of the number of inflows, outflows, and real-time population in the region, the number of permanent residents, passers-by, and tourists in the current population of the region, and the distribution of the current population's source, direction of entry into the region, and direction of exit from the region. Among them, the definition of regional population is as follows: Figure 3 As shown; the definition of the current population type of the region is as follows Figure 4 As shown; the schematic diagram of the current population origin calculation is as follows Figure 5 As shown;

[0012] Regional population flow forecasting: mainly refers to studying the historical changes in the number of people flowing into the region, and predicting the future number of inflows, outflows, and real-time population in the region.

[0013] Furthermore, in the real-time calculation of regional human flow, considering that individual behavior should be sequential, persistent, and continuous, and that individual behavior is uncertain, a Markov chain dynamic and static state judgment model based on the movement of people in the region is established according to the above characteristics of the individual (such as Figure 2 Calculate individual movement status in chronological order and analyze the flow of people in the area in real time;

[0014] The historical movement state of an individual in the region can be expressed as:

[0015] A random process {X n ,n∈T}, and for any time granularity t i The probability P{X n+1 =i n+1 |X0=i0,X1=i1,…,X n =i n}=P{X n+1 =i n+1 |X n =i n} state transfer process, which can determine the individual's movement and stillness at present and in the future based on the historical movement state, and thus determine whether the individual has left the area.

[0016] Specifically, the production end realizes the real-time access function of spatiotemporal information data, and further comprises the following steps:

[0017] 1) In order to solve the problem of data integrity and timeliness in the context of big data and meet the computing requirements, a task scheduling mechanism, task scheduling algorithm and buffer pool mechanism based on Spark AMC are proposed to act on the data quality judgment and task scheduling initialization of the production end of Spark streaming real-time tasks;

[0018] 2) The real-time computing framework based on the big data platform will have a minimum update time granularity t i , the amount of data expected to be received within the current time granularity Actual amount of data received To measure the data quality i , establish a buffer pool R, generate a scheduling sequence array Set the value in the array to -999;

[0019] The numbers represent the following meanings:

[0020] Ready state; Waiting state; Completion status; Resend status;

[0021] The algorithm of data evaluation and task scheduling mechanism is:

[0022]

[0023] 3) Completed data quality i Initialization of scheduling algorithms in judgment and task scheduling ensures data integrity and timeliness.

[0024] Preferably, the consumer end reads the cache pool data, determines the data quality, and generates a schedule for high-quality data in real time, further comprising the following steps:

[0025] 1) For the data that does not meet the requirements temporarily, merge it with the currently accessed data to perform calculation and schedule algorithm design: i The packets are grouped and stored in the buffer pool R; at this time, the scheduling sequence is in a blocked state, that is: Waiting state, remember to press t in R in the buffer pool i The data set to be grouped Compare the data in the buffer pool with the data in the current data stream The following scheduling algorithm is used for merging buffer pools:

[0026]

[0027] 2) For the scheduling sequence when Execute the spark task, and wait or resend in other cases; the whole set of task scheduling mechanism algorithms, as well as the data supplement work, are finally packaged into a spark environment where the amc task scheduling scheme runs automatically;

[0028] 3) If the current scheduling task is in When the state is reached, the prediction task is accessed in real time; under the guarantee of the above-mentioned scheduling mechanism, the data quality and data continuity of the real-time data sent are fully guaranteed, so that the accuracy and timeliness of the real-time calculation of the number of people flow are fully guaranteed.

[0029] Further preferably, the real-time calculation of regional human flow also includes:

[0030] 1) Establish individual portraits and calculate multiple labels of people, including: daytime residence, nighttime residence, and origin; among which:

[0031] Daytime permanent location: Extract the movement status of the individual's daytime temporal behavior, analyze the location and duration of each person's stay, and continuously count for a period of time to analyze the individual's daytime permanent location;

[0032] Nighttime permanent location: extract the movement status of the individual's nighttime temporal behavior and analyze each individual's nighttime permanent location; origin: calculate the user's origin based on the user's historical call records, which may be their hometown or main business location;

[0033] 2) In real-time task calculation, considering that individual behavior should be sequential, persistent, and continuous, a Markov model is established for each person in the area, and the individual state is calculated in chronological order to analyze the flow of people in the area in real time;

[0034] 3) Real-time calculation of the number of inflow and outflow population in the region and the real-time population;

[0035] 4) Calculate the number of permanent residents, passers-by, and tourists in the current population of the region, and calculate the distribution of the current population's origin, the direction of entry into the region, and the direction of leaving the region.

[0036] The regional passenger flow prediction also includes:

[0037] 1) Based on the spatiotemporal data of regional historical population flow, the long short-term memory network algorithm in the deep neural network is used to generate time series X based on the historical data of regional population flow. k ={x1,x2,x3,…,x k} as input to build a real-time crowd flow prediction model;

[0038] 2) The real-time crowd flow prediction model uses the basic unit of LSTM network (such as Figure 7As shown in Figure 2, the basic network unit is k As input, a memory network unit is obtained by value transfer calculation in its three-layer structure, and finally the whole network is fully connected using the softmax function to obtain a prediction of each time granularity t in the future period based on historical population flow data. i Here we briefly describe the operation mechanism of the three-layer structure of LSTM. In LSTM, the first layer determines the input sequence X k Which information should be retained and which information should be forgotten? This layer of structure is called the forget gate, where:

[0039] Forget gate: The forget gate is the output of the previous layer and the sequence data X to be input to this layer. k As input, through an activation function sigmoid, the output is At this time Indicates the probability that the previous layer unit is forgotten, where 1 means "completely retained" and 0 means "completely discarded";

[0040]

[0041] 3) Enter the second layer again and process the new input data X t Processing is performed to determine the new input data that needs to be retained and injected into the computing unit. This layer of structure is called the input gate, where:

[0042] The input gate consists of two parts. The first part uses the sigmoid activation function and the output is The second part uses the tanh activation function and the output is Indicates the degree to which data is retained. Indicates the degree to which the information of this layer is retained;

[0043] i t =δ(M i .[h t-1 ,x t ]+b i )

[0044]

[0045] 4) After the data is filtered by the first-layer forget gate and the second-layer input gate updates the newly input data, the retention degree information of the newly input data can be obtained. Based on this, the newly input information can be updated to the unit state C of this layer t ;

[0046]

[0047] Finally, the value is passed to the output gate;

[0048] The output gate is used to control the filtering of the unit state of this layer. First, the sigmoid activation function is used to obtain an o t [0,1] interval, and then the unit state C t After being processed by the tanh activation function and multiplied by , the output h of this layer is obtained. t ;

[0049] o t =δ(M o [h t-1 ,x t ]+b o )

[0050] h t =o t *tanh(C t )

[0051] 5) Then we select long-term historical data and use the LSTM network to predict the trend of the next few days or weeks to get the output sequence L i ={l1,l2,l3,…,l i}, make reasonable predictions about future trends;

[0052] 6) Using bidirectional LSTM (network structure such as Figure 6 As shown in the figure, short-term prediction is performed at t i Make predictions within the cycle, achieve real-time prediction completion, and provide prediction supplements when tasks are blocked.

[0053] The real-time monitoring and calculation method of urban regional pedestrian flow in the present invention actually solves the problem of real-time calculation of tens of millions of moving objects based on spatiotemporal big data, has the advantages of highly flexible streaming time window, high throughput, etc., the calculation results are more accurate, supports offline supplementary recording results, and provides regional pedestrian flow prediction function within a certain time range. Compared with the existing technology, the beneficial technical effects brought are:

[0054] (1) In public places with high population density in cities, such as scenic spots, shopping malls, stadiums, conference halls, etc., real-time understanding of regional passenger flow conditions can be achieved, and future passenger flow trends can be predicted to provide timely and effective support for relevant work of government departments;

[0055] (2) A solution to the real-time calculation problem of tens of millions of moving objects based on spatiotemporal big data, which effectively compensates for the inaccurate business results caused by incomplete data. It has the advantages of highly flexible streaming time windows and high throughput, more accurate calculation results, supports offline supplementary recording results, and provides regional passenger flow prediction function within a certain time range;

[0056] (3) Supports monitoring of any area, and can realize monitoring and analysis of the area by obtaining the area boundary data;

[0057] (4) It has a high promotion value and can be applied in the military field, the people's livelihood field and other fields to bring about a more extensive and revolutionary impact. It has a forward-looking nature in the essential theories, methods and technologies for solving the current challenges. It can realize the transformation from "data-information-knowledge-decision support" to "data-knowledge-decision support" and form a product system with software products, software and hardware integrated products and data products as the core. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a framework diagram of the calculation method for real-time monitoring of regional human flow of the present invention;

[0059] Figure 2 Schematic diagram of the individual state calculation process;

[0060] Figure 3 Schematic diagram of the cache pool to solve the problem of incomplete data in real-time business scenarios;

[0061] Figure 4 This is a schematic diagram of the population behavior state recognition algorithm;

[0062] Figure 5 This is a schematic diagram of the population origin algorithm (communication cube);

[0063] Figure 6 To describe the schematic diagram of the bidirectional LST network structure;

[0064] Figure 7 To describe the schematic diagram of the basic unit structure of LSTM;

[0065] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0066] See also Figure 1 This embodiment provides a method for real-time monitoring and calculation of urban area pedestrian flow, including the following steps:

[0067] Step 1: Obtain the boundary information of any area in the city and calculate the situation of the area;

[0068] Step 2: Divide the acquired boundary information of any area in the city into production end, consumption end, and task supplementary recording mechanism;

[0069] Step 3: On the production side, it mainly realizes the real-time access function of spatiotemporal information data, completes data cleaning, judges data quality, generates real-time schedules for high-quality data, extracts features for low-quality data, generates intermediate tables, and stores them in the data cache pool;

[0070] Step 4: On the consumer side, read the cache pool data, determine the data quality, and generate a schedule for high-quality data in real time; perform task scheduling based on the schedule to mark the task status, execution order, restart times, etc. Task scheduling mainly includes computing tasks and prediction tasks;

[0071] Step 5: Re-record the task, read offline data, determine data quality, and calculate the flow of people in the area;

[0072] Step 6: Real-time calculation of regional human flow, mainly refers to real-time calculation of the number of inflows, outflows, and real-time population in the region, calculation of the number of permanent residents, passers-by, and tourists in the current population of the region, calculation of the distribution of the current population's source, the direction of entry into the region, and the direction of leaving the region.

[0073] Step 7: Regional passenger flow prediction mainly refers to studying the historical changes in the number of passengers in the region, and predicting the future number of inflows, outflows, and real-time population in the region.

[0074] In this embodiment, the real-time monitoring calculation method in step 2 includes three parts. The production end and the consumption end mainly complete the real-time calculation and prediction functions, and the task supplement completes the historical data calculation function. In the real-time task calculation, considering that individual behavior should be sequential, persistent, and continuous, and at the same time, the individual behavior is uncertain, a Markov chain dynamic and static state judgment model based on the movement of people in the area is established according to the above characteristics of the individual. Calculate the individual movement state in chronological order and analyze the flow of people in the area in real time;

[0075] Step 2.1, the historical movement state of an individual in the region can be expressed as:

[0076] A random process {X n ,n∈T}, and for any time granularity t i The probability P{X n+1 =i n+1 |X0=i0,X1=i1,…,X n =i n}=P{X n+1 =i n+1 |X n =i n} state transfer process, which can determine the individual's movement and stillness at present and in the future based on the historical movement state, and thus determine whether the individual has left the area.

[0077] Furthermore, step 3 also includes the following sub-steps:

[0078] Step 3.1, in order to solve the integrity and timeliness of data in the context of big data and meet the computing requirements, a task scheduling mechanism, task scheduling algorithm and buffer pool mechanism based on Spark AMC are proposed to act on the data quality judgment and task scheduling initialization of the production end of Spark streaming real-time tasks;

[0079] Step 3.2: The real-time performance of the real-time computing framework based on the big data platform will have a minimum update time granularity t i , the amount of data expected to be received within the current time granularity Actual amount of data received To measure the data quality i , establish a buffer pool R, generate a scheduling sequence array Set the value in the array to -999;

[0080] The numbers represent the following meanings:

[0081] Ready state; Waiting state; Completion status; Retransmission status; the data evaluation and task scheduling mechanism algorithm is:

[0082]

[0083] Step 3.3, completed the data quality i The initialization of the scheduling algorithm in judgment and task scheduling ensures the integrity and timeliness of the data.

[0084] In this embodiment, step 4 includes the following sub-steps:

[0085] Step 4.1: For the data that does not meet the requirements temporarily, merge it with the currently accessed data, and perform calculation and scheduling algorithm design: i The packets are grouped and stored in the buffer pool R. At this time, the scheduling sequence is in a blocked state: Waiting state, remember to press t in R in the buffer pool i The data set to be grouped Compare the data in the buffer pool with the data in the current data stream The following scheduling algorithm for the buffer pool is used for merging.

[0086]

[0087] Step 4.2, for the scheduling sequence when Execute the spark task, and wait or resend in other cases. The entire task scheduling mechanism algorithm also completes the data supplement work, and finally encapsulates it into a spark environment where the amc task scheduling solution runs automatically.

[0088] Step 4.3, if the current scheduled task is in When the state is reached, the prediction task is accessed in real time. Under the guarantee of the above scheduling mechanism, the data quality and data continuity of the real-time data are fully guaranteed, so that the accuracy and timeliness of the real-time calculation of the number of people flow are fully guaranteed; it solves the pain points and difficulties raised by this patent. At the same time, a real-time task scheduling system for the data platform is also established.

[0089] Preferably, step 6 includes the following sub-steps:

[0090] Step 6.1: Create an individual portrait and calculate multiple labels of the person, including: daytime permanent location, nighttime permanent location, and origin. Daytime permanent location: extract the movement status of the individual's daytime temporal behavior, analyze the location and duration of each person's stay (cluster the stay location through relevant clustering algorithms combined with spatiotemporal data characteristics), continuously count for a period of time, and analyze the individual's daytime permanent location; nighttime permanent location: extract the movement status of the individual's nighttime temporal behavior, and analyze the nighttime permanent location of each individual; origin: calculate the user's origin based on the user's historical call records. The origin may be his hometown, main business location, etc.;

[0091] Step 6.2: In real-time task calculation, considering that individual behavior should be sequential, persistent, and continuous, the present invention establishes a Markov model for each person in the area, calculates individual states in chronological order, and analyzes the flow of people in the area in real time;

[0092] Step 6.3: Calculate the number of inflow population, outflow population and real-time population in the region in real time;

[0093] Step 6.4: Calculate the number of permanent residents, passers-by, and tourists in the current population of the region, and calculate the distribution of the current population's origin, the direction of entry into the region, and the direction of leaving the region.

[0094] And the sub-steps of step 7:

[0095] Step 7.1: Based on the spatiotemporal data of regional historical passenger flow, the Long Short-Term Memory (LSTM) algorithm in the deep neural network is used to implement the time series X based on historical data. k ={x1,x2,x3,…,x k} as input to build a real-time crowd flow prediction model.

[0096] Step 7.2: The real-time crowd flow prediction model uses the basic unit of the LSTM network (such as Figure 7 As shown in the figure, it solves the problem that traditional RNN (recurrent neural network) cannot connect the context. In LSTM, the first stage is the forget gate, and the forget layer determines which input information should be forgotten.

[0097] Step 7.2.1 Forget gate: The forget gate is the output of the previous layer and the sequence data x to be input to this layer. t As input, through an activation function sigmoid, the output is f t .f t The output value of is in the range of [0,1], indicating the probability that the previous layer unit is forgotten, 1 means "completely retained" and 0 means "completely discarded".

[0098] f t =δ(M f .[h t-1 ,x t ]+b f )

[0099] Step 7.3 enters the input gate of the next stage, which determines what new information needs to be placed in the unit.

[0100] Step 7.3.1 Input gate: The input gate consists of two parts. The first part uses the sigmoid activation function and the output is The second part uses the tanh activation function and the output is It is expressed as the degree to which information is retained. Indicates the degree to which the information of this layer is retained.

[0101] i t =δ(M i .[h t-1 ,x t ]+b i )

[0102]

[0103] Step 7.4: Now we know that f t It is the output of the forget gate, which controls the state C of the previous unit. t-1 The degree of being forgotten, Indicates how much new information is retained, based on which the newly input information can be updated to the state C of this layer unit t

[0104]

[0105] Finally the value is passed to the output gate.

[0106] Step 7.4.1 Output gate: The output gate is used to control the filtering of the unit state of this layer. First, use the sigmoid activation function to get an o t [0,1] interval, and then the unit state C t After being processed by the tanh activation function and multiplied by , the output h of this layer is obtained. t .

[0107] o t =δ(M o [h t-1 ,x t ]+b o )

[0108] h t =o t *tanh(C t )

[0109] Step 7.5: Select long-term historical data and use the LSTM network to predict the trend of the next few days or weeks to obtain the output sequence L i ={l1,l2,l3,…,l i}, make reasonable predictions about future trends.

[0110] Step 7.6: Use bidirectional LSTM to make short-term predictions at t i Make predictions within the cycle, achieve real-time prediction completion, and provide prediction supplements when tasks are blocked.

[0111] The network structure is as follows Figure 3 As shown in the figure, the bidirectional neural network is composed of two LSTMs that propagate forward and backward. A forward LSTM uses past information, and a reverse LSTM uses future information. In this way, at time t, both the information at time t-1 and the information at time t+1 can be used, which makes the result more accurate.

[0112] The following are examples given by the inventors.

[0113] Example:

[0114] Figure 1 The production and consumption process is built based on hadoop 2.6.0+spark 1.6.0+Flume 1.2.0+Spark Streaming+Kafka environment.

[0115] 1) The production end pushes data through Spark streaming real-time streams and opens windows with a minimum time granularity of five minutes. It can receive five minutes of real-time data, clean the data, group it according to the minimum time granularity time interval, and judge the quality of each group of data. The data that meets the threshold condition of 80% will be passed to the consumer end for consumption;

[0116] 2) Buffer pool module: For data that does not meet the threshold condition of 80%, it is stored in the partition table partitioned by time granularity in Hive. When the next real-time stream is connected, the data in the partition table is merged to jointly determine whether the data quality meets the threshold. If it still does not meet the threshold condition, it continues to be stored in the current partition table, and the maximum storage time is set to 30 minutes (that is, 6 time granularity cycles).

[0117] 3) Task scheduling system module: A buffer pool mechanism is set up on the production side to ensure data quality, but it also brings about the problem of task scheduling for consumption data on the consumer side. To solve this problem, a task scheduling sequence is established. When the data quality is satisfied, the current time granularity t i Corresponding Assigning the value to the ready state is: When the data quality does not meet the threshold condition, the state is assigned to the waiting state, that is: When the completion of t i When consuming data, it is assigned to the completed state, that is: When an error occurs in the task being consumed and needs to be resent, its value remains unchanged. There are the following scheduling rules: For the scheduling sequence Need to follow the order, when there are six consecutive When the first The value of The sequence should now be In the past, the transition was completed.

[0118] 4) Prediction module: When the task status is in the waiting state, the entire real-time flow will stagnate. In order to solve this problem, an AMC scheduling system is established. When the entire system is in the waiting state, it will immediately switch to the prediction system.

[0119] The historical population flow data of the region is counted at five-minute intervals to create a time series data set (in t iThe index is the population flow, and the population flow is the value). On the platform of Tensorflow 2.0+python3.7+centos11.0+1080Ti 8G×4, the time series of one year is used as the training set to input the LSTM network and the bidirectional LSTM network to train the relevant models, and then the current time t i As input, the current regional flow of people is predicted. Finally, it is pushed to the AMC system through the REST service.

[0120] 5) Consumer module: Real-time calculation of the number of people flowing into the region, the number of people flowing out, and the real-time number of people; calculation of the number of permanent residents, passers-by, and tourists in the current population of the region, and calculation of the distribution of the current population's source, the direction of entry into the region, and the direction of leaving the region.

Claims

1. A method for real-time monitoring and calculation of pedestrian flow in urban areas, characterized in that: This method divides the obtained boundary information of any area in the city into production end, consumption end, and task supplement, where: The production side mainly implements the real-time access function of spatiotemporal information data, completes data cleaning, judges data quality, generates real-time schedules for high-quality data, extracts features for low-quality data, generates intermediate tables, and stores them in the data cache pool; The consumer side reads the cache pool data, determines the data quality, generates a schedule for high-quality data in real time, and schedules tasks based on the task status, execution order, and restart times marked in the schedule. Task scheduling mainly includes computing tasks and prediction tasks. Re-record tasks, read offline data, judge data quality, and calculate regional passenger flow; The calculation of regional crowd flow is divided into regional crowd flow real-time calculation and regional crowd flow prediction; wherein: Real-time calculation of regional human flow: mainly refers to the real-time calculation of the number of inflows, outflows, and real-time population in the region, the number of permanent residents, passers-by, and tourists in the current population of the region, the distribution of the current population's source, the distribution of the direction of entry into the region, and the distribution of the direction of leaving the region; Regional population flow prediction: mainly refers to studying the changing patterns of regional historical population flow, predicting the future inflow and outflow of population, and the real-time population of the region; The production end realizes the real-time access function of spatiotemporal information data, and also includes the following steps: 1) In order to solve the integrity and timeliness of data in the context of big data and meet the computing requirements, a task scheduling mechanism, task scheduling algorithm and buffer pool mechanism based on Spark AMC are proposed to act on the data quality judgment and task scheduling initialization of the production end of Spark streaming real-time tasks; 2) The real-time computing framework based on the big data platform will have a minimum update time granularity t i , the expected amount of data received within the current time granularity N ti 、Actual received data volume n ti ; To measure data quality r i , establish a buffer pool R, generate a scheduling sequence array Set the value in the array to -999; The numbers in the table represent the following meanings: Ready state; Waiting state; Completion status; Resend status; The algorithm of data evaluation and task scheduling mechanism is: 3) Completed data quality i Initialization of scheduling algorithms in judgment and task scheduling to ensure data integrity and timeliness; The consumer end reads the cache pool data, determines the data quality, and generates a schedule for high-quality data in real time, and also includes the following steps: 1) For the data that does not meet the requirements temporarily, merge it with the currently accessed data to perform calculation and schedule algorithm design: i The groups are stored in the buffer pool R; at this time, the scheduling sequence is in a blocked state, that is: a ti =-1 waiting state, record in the buffer pool R press t i The data set to be grouped Compare the data in the buffer pool with the data in the current data stream The following scheduling algorithm is used for merging buffer pools: 2) For the scheduling sequence when Execute the spark task, wait or resend in other cases; complete the data supplement work while implementing the whole set of task scheduling mechanism algorithm, and finally encapsulate it into the spark environment where the amc task scheduling scheme runs automatically; 3) If the current scheduling task is in When the state is reached, the prediction task is accessed in real time; under the guarantee of the above scheduling mechanism, the data quality and data continuity of the real-time data sent are fully guaranteed, so that the accuracy and timeliness of the real-time calculation of the number of people flow are fully guaranteed; The real-time calculation of regional human flow also includes: 1) Establish individual portraits and calculate multiple labels of people, including: daytime residence, nighttime residence, and origin; among which: Daytime permanent location: Extract the movement status of the individual's daytime temporal behavior, analyze the location and duration of each person's stay, and continuously count for a period of time to analyze the individual's daytime permanent location; Nighttime permanent location: extract the movement status of the individual's nighttime temporal behavior and analyze each individual's nighttime permanent location; origin: calculate the user's origin based on the user's historical call records, which may be their hometown or main business location; 2) In real-time task calculation, considering that individual behavior should be sequential, persistent, and continuous, a Markov model is established for each person in the area, and the individual state is calculated in chronological order to analyze the flow of people in the area in real time; 3) Real-time calculation of the number of inflow and outflow population in the region and the real-time population; 4) Calculate the number of permanent residents, passers-by, and tourists in the current population of the region, and calculate the distribution of the current population's origin, the direction of entry into the region, and the direction of leaving the region; The regional passenger flow prediction also includes: 1) Based on the spatiotemporal data of regional historical population flow, the long short-term memory network algorithm in the deep neural network is used to generate time series X based on the historical data of regional population flow. k ={x1,x2,x3,…,x k } as input to build a real-time crowd flow prediction model; 2) The real-time crowd flow prediction model uses the basic unit of the LSTM network as the basic network unit, and uses the time series X k As input, a memory network unit is obtained by value transfer calculation in its three-layer structure, and finally the whole network is fully connected using the softmax function to obtain a prediction of each time granularity t in the future period based on historical population flow data. i The prediction of population flow; Here we briefly describe the operation mechanism of the three-layer structure of LSTM; In LSTM, the first layer determines the input sequence X k Which information should be retained and which information should be forgotten? This layer of structure is called the forget gate, where: Forget gate: The forget gate is the output of the previous layer and the sequence data X to be input to this layer. k As input, through an activation function sigmoid, the output is At this time Indicates the probability that the previous layer unit is forgotten, where 1 means "completely retained" and 0 means "completely discarded"; 3) Enter the second layer again and process the newly input data X t Processing is performed to determine the new input data that needs to be retained and injected into the computing unit. This layer of structure is called the input gate, where: The input gate consists of two parts. The first part uses the sigmoid activation function and the output is The second part uses the tanh activation function and the output is Indicates the degree to which data is retained. Indicates the degree to which the information of this layer is retained; i t =δ(M i .[h t-1 ,x t ]+b i ) 4) After the data is filtered by the forget gate of the first layer and the new input data is updated by the input gate of the second layer, the retention degree information of the new input data is obtained Based on this, the new input information is updated to the unit state C of this layer t ; Finally, the value is passed to the output gate; The output gate is used to control the filtering of the unit state of this layer. First, the sigmoid activation function is used to obtain an o t [0,1] interval, and then the unit state C t After being processed by the tanh activation function and multiplied by , the output h of this layer is obtained. t ; o t =δ(M o [h t-1 ,x t ]+b o ) h t =o t *tanh(C t ) 5) Then we select long-term historical data and use the LSTM network to predict the trend of the next few days or weeks to get the output sequence L i ={l1,l2,l3,…,l i }, make reasonable predictions about future trends; 6) Use bidirectional LSTM for short-term prediction at t i Make predictions within the cycle, achieve real-time prediction completion, and provide prediction supplements when tasks are blocked.

2. The method according to claim 1, characterized in that In the real-time calculation of regional human flow, considering that individual behavior should be sequential, persistent, and continuous, and at the same time, individual behavior is uncertain, a Markov chain dynamic and static state judgment model based on the movement of people in the region is established according to the above characteristics of individuals; the individual movement state is calculated in chronological order, and the human flow situation in the region is analyzed in real time; The historical motion state of an individual in the region is expressed as: A random process {X n ,n∈T}, and for any time granularity t i The probability P{X n+1 =i n+1 |X0=i0,X1=i1,…,X n =i n }=P{X n+1 =i n+1 |X n =i n } state transfer process, which completes the judgment of the individual's movement and stillness at present and future moments based on the historical movement state, so as to judge whether the individual has left the area.

Images