Three-layer network passenger flow parameter calculation method and calibration system

Abstract

The invention provides a three-layer network passenger flow parameter calculation method and a calibration system. The three-layer network passenger flow parameter calculation method comprises the following steps: calibrating a to-be-detected camera by using a calibration device and performing screenshot on the to-be-detected camera; determining a coordinate conversion model and regional parameters by using a parameter calibration system in combination with the screenshot icon, and storing calibration parameters and corresponding camera numbers; the method comprises the following steps: reading a configuration file of a camera to be detected, reading a camera code stream, performing hardware decoding on the video code stream, and extracting frames to store a queue; reading video frames in the queue, detecting and tracking pedestrians to generate pedestrian id, and binding head pixel coordinate information, age information and id; according to different age information of pedestrians, head pixel coordinate information is substituted into coordinate conversion models of different heights, and the actual spatial position is calculated; according to the actual space position, back calculation is carried out in the area, corresponding passenger flow parameters are calculated, and a result is output. According to the method and the system provided by the invention, calibration of two-dimensional space coordinate systems at different heights and detection and calculation of real-time passenger flow parameters are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of rail transportation, and more specifically, relates to a three-layer network passenger flow parameter calculation method and calibration system. Background Art

[0002] There are many cities in my country with large subway passenger flows. In order to solve the problem of large passenger flows caused by transportation, tourist attractions, and large-scale events, video detection technology is usually used to monitor the size of passenger flows.

[0003] The calibration method of the existing commonly used video detection technology usually uses ground reference objects to calibrate the ground two-dimensional space coordinate system, and then captures the camera's picture to calculate the internal parameters. The existing technology has the problem that most cameras in actual scenes are fixed angles, lack of multi-angle and multi-position camera screenshots, and the calculation results have large errors. Summary of the invention

[0004] The purpose of the present invention is to provide a three-layer network passenger flow parameter calculation method and calibration system, aiming to solve the technical problem that the calculation results in the prior art have large errors.

[0005] To achieve the above object, the technical solution adopted by the present invention is: to provide a three-layer network passenger flow parameter calculation method, comprising the following steps:

[0006] Use the calibration device to calibrate the camera to be tested and take a screenshot;

[0007] Use the parameter calibration system combined with the screenshot to calibrate the coordinate transformation model and area parameters, and save the calibration parameters and the corresponding camera number;

[0008] Read the configuration file of the camera to be detected, read the camera code stream, perform hardware decoding on the video code stream, extract frames and store them in a queue; then read the video frames in the queue, generate a pedestrian ID for pedestrian detection and tracking, and bind the head pixel coordinate information and age information to the ID;

[0009] According to the different age information of pedestrians, the head pixel coordinate information is brought into the coordinate conversion model at different heights to calculate the actual spatial position;

[0010] According to the actual spatial position, the calculation is reversed to the area, the corresponding passenger flow parameters are calculated and the results are output.

[0011] Preferably, the use of the parameter calibration system in combination with the screenshot to calibrate the coordinate transformation model and the area parameters, and save the calibration parameters and the corresponding camera number, includes:

[0012] Capture multiple sets of calibration plate images at different positions and angles under the camera; identify the corner points of the calibration plate in the image; calculate the camera intrinsic parameter matrix and distortion coefficient; and generate the correction image and the new internal parameter after correction;

[0013] Perform scene calibration, use the calibration cloth to capture pictures according to the average height, and then mark multiple pixel coordinate reference points in the screenshots;

[0014] Generate a two-dimensional world coordinate system according to the actual scene; identify the world coordinate system corresponding to the corner coordinates of the calibration cloth in the coordinate system; store the world coordinates of the calibration cloth;

[0015] Solve for the external parameters for the transformation.

[0016] Preferably, the step of bringing the head pixel coordinate information into coordinate conversion models at different heights according to different age information of pedestrians to calculate the actual spatial position comprises:

[0017] According to the age information of the identified pedestrian, the pixel coordinates of the pedestrian's head are converted to the corrected pixel coordinates of the head through the camera's internal parameters. The camera distortion coefficient is known. For any point (x, y) in the image, the distortion correction is performed through the inverse operation of the above distortion formula. The corrected coordinates are:

[0018] The same is true for tangential distortion;

[0019] The corrected head pixel coordinates are brought into the coordinate transformation model of the corresponding age, and transformed into the head two-dimensional space coordinate system, that is, the actual space coordinates, according to the homography matrix; the transformation formula is:

[0020] u i =h 11 x i +h 12 y i +h 13 , v i =h 21 x i +h 22 y i +h 23 ,ω i =h 31 x i +h 32 y i +h 33 .

[0021] Preferably, the method of capturing multiple groups of calibration plate images at different positions and angles under the camera; identifying the corner points of the calibration plate in the image; calculating the camera intrinsic parameter matrix and distortion coefficient; and generating a correction image and a new corrected intrinsic parameter includes:

[0022] Assume that the coordinates of the corner points of the calibration plate in the spatial coordinate system are (X a ,Y a ,Z a ) (Usually the calibration plate plane is placed in the spatial coordinate system Z a =0), the coordinates in the camera coordinate system are (X b ,Y b ,Z b ), the transformation relationship from the space coordinate system to the camera coordinate system is represented by the rotation matrix R and the translation vector t, that is,

[0023] Preferably, the method of capturing multiple groups of calibration plate images at different positions and angles under the camera; identifying the corner points of the calibration plate in the image; calculating the camera intrinsic parameter matrix and distortion coefficient; and generating a correction image and a new corrected intrinsic parameter includes:

[0024] The tangential distortion formula is:

[0025] x 畸变 =x+(2*p1*x*y+p2*(r 2 +2*x 2 )),y 畸变 =y+(p1*(r 2 +2*y 2 )+2*p2*x*y) where p1, p2 are tangential distortion coefficients; r 2 =x 2 +y 2 .

[0026] The radial distortion formula is:

[0027] x 畸变 =x(1+k1*r 2 +k2*r 4 +k3*r 6 ), y 畸变 =y(1+k1*r 2 +k2*r 4 +k3*r 6 ), where k1, k2, and k3 are radial distortion coefficients.

[0028] Distortion correction: Given a known distortion coefficient, for any point (x, y) in the image, distortion correction is performed by inverse calculation of the above distortion formula. The corrected coordinates are:

[0029]

[0030] Preferably, the method of capturing multiple groups of calibration plate images at different positions and angles under the camera; identifying the corner points of the calibration plate in the image; calculating the camera intrinsic parameter matrix and distortion coefficient; and generating a correction image and a new corrected intrinsic parameter includes:

[0031] Assume the coordinates in the pixel coordinate system are (u, v), and introduce the focal length f χ 、f y , image center coordinates (u0, v0) and pixel offset coefficient c, the conversion relationship is where s χ and y The number of pixels per unit length.

[0032] Preferably, the method of capturing multiple sets of calibration plate images at different positions and angles under the camera; identifying the corner points of the calibration plate in the image; calculating the camera intrinsic parameter matrix and distortion coefficient; and generating a correction image and a new internal parameter after correction includes: using an optimization algorithm such as the least squares method to solve the intrinsic parameter matrix The various parameters in .

[0033] Preferably, the back-calculation into the area based on the actual spatial position, calculating the corresponding passenger flow parameters and outputting the results includes: the area includes one or more of a density area, a speed area, a queuing area, and an out-of-bounds area.

[0034] Preferably, the configuration file includes: one or more of: a gpu index, a camera number, a code stream address, a detection model address, a coordinate conversion model parameter and an area parameter.

[0035] The present invention also provides a calibration system for calculating passenger flow parameters of a three-layer network, characterized in that it is used in the steps of a three-layer network passenger flow parameter calculation method as described in any one of the above items, and comprises: a first calibration component and a second calibration component;

[0036] The first calibration component includes a control structure, a calibration bracket, a first motor, a second motor, a third motor, a calibration board, and a walking structure; the bottom end of the calibration bracket is connected and fixed to the walking structure; the top end of the calibration bracket is connected and fixed with the first motor; the first motor is connected to the second motor through a first transmission rod; the second motor is connected to the third motor through a second transmission rod; the third motor is connected and fixed to the calibration board;

[0037] The second calibration component includes a remote control, a structural support, a fourth motor, a fifth motor, a sixth motor, a seventh motor, an eighth motor, a ninth motor calibration cloth and a second walking structure.

[0038] The beneficial effects of the three-layer network passenger flow parameter calculation method and calibration system provided by the present invention are as follows: Compared with the prior art, the three-layer network passenger flow parameter calculation method and calibration system provided by the present invention, in the calibration process, the average height of each stage is divided into three dimensions of children, teenagers, and adults, and the average height is used as a two-dimensional space coordinate system, that is, a two-dimensional space coordinate system of the head, and the relevant calibration tools and parameter calibration schemes are customized to calibrate the conversion parameters of the pixel coordinates and the two-dimensional space coordinate system of the head and the regional parameters based on the two-dimensional space coordinate system of the head, including parameters such as effective density area, speed area, out-of-bounds area, and queue area. In target detection, first detect whether it is a child, teenager or adult, and then use the pixel coordinates of the person's head to convert them into the two-dimensional space coordinate system of the head of children, teenagers and adults through the conversion parameters to calculate their actual positions, and bring in the regional parameters according to the actual position to further calculate multiple passenger flow parameters including personnel density, speed, queue length, duration, whether out-of-bounds abnormal behavior occurs, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A flowchart of a method for calculating passenger flow parameters in a three-layer network provided by an embodiment of the present invention; Figure 2 A flowchart of step S1 in a three-layer network passenger flow parameter calculation method provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a first calibration component used in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a second calibration component used in an embodiment of the present invention; Figure 5 Schematic diagram of different usage states of the calibration cloth of the second calibration component used in the embodiment of the present invention; Figure 6 A schematic diagram of a three-layer network conversion process in a three-layer network passenger flow parameter calculation method provided by an embodiment of the present invention; Figure 7 It is the heat map of the 2D spatial coordinates of the pedestrian’s head; Figure 8 Calculate the flow chart for pedestrian speed; Fig. 9 Flowchart for pedestrian crossing alarm. DETAILED DESCRIPTION

[0041] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] Existing calibration methods are usually based on the ground coordinate system. The calculation of the person's position and speed is based on the conversion of the pixel coordinates of the soles of the feet into the ground two-dimensional space coordinate system. This method has no problem with the calculation accuracy when there are few people. However, in reality, many scenes are densely populated, and the feet of most people are blocked in the camera screen, making it impossible to calculate the person's position and walking speed by identifying the feet.

[0043] The present invention provides a three-layer network passenger flow parameter calculation method. Aiming at the problem that there are large errors in the calculation of passenger flow parameters in the monitoring screen of dense crowds in the prior art, a method for calculating passenger flow parameters in dense crowds is provided. In the calibration process, the average height of each stage is divided into three dimensions: the average height of children, teenagers, and adults. The average height is used as a two-dimensional space coordinate system, that is, a two-dimensional space coordinate system of the head. Relevant calibration tools and parameter calibration schemes are customized to calibrate the conversion parameters of pixel coordinates and the two-dimensional space coordinate system of the head and the regional parameters based on the two-dimensional space coordinate system of the head, including parameters such as effective density area, speed area, out-of-bounds area, and queue area. In target detection, first detect whether it is a child, teenager, or adult, and then use the pixel coordinates of the person's head to convert them into the two-dimensional space coordinate system of the head of children, teenagers, and adults through the conversion parameters to calculate their actual positions. According to the actual position, the regional parameters are brought in to further calculate multiple passenger flow parameters including personnel density, speed, queue length, duration, whether out-of-bounds abnormal behavior occurs, etc. For details, please refer to Figures 1 to 9 Now, a three-layer network passenger flow parameter calculation method provided by the present invention is described. The three-layer network passenger flow parameter calculation method comprises the following steps:

[0044] S1. Use a calibration device to calibrate the camera to be tested and take a screenshot;

[0045] Preferably, step S1 includes the following steps: S1.1, assembling a calibration device; the calibration device includes a first calibration component and a second calibration component.

[0046] The first calibration component: includes a calibration bracket, a calibration plate set on the bracket, and a driving motor; the height of the calibration bracket and the angle of the calibration plate can be adjusted by Bluetooth remote control to meet the needs of intercepting multiple sets of calibration plate images at different positions and angles; the calibration bracket is a 7-meter-high telescopic bracket. This set of equipment is divided into two parts: a remote control and a calibration device. The left remote control and the right calibration device are connected via Bluetooth. The first calibration component includes a control structure, a calibration bracket 5, a first motor 1, a second motor 2, a third motor 3, a calibration plate 4, and a walking structure 6. The bottom end of the calibration bracket 5 is connected and fixed to the walking structure 6. The top of the calibration bracket 5 is connected and fixed with a first motor 1. The first motor 1 is connected to the second motor 2 through a first transmission rod. The second motor 2 is connected to the third motor 3 through a second transmission rod. The third motor 3 is connected and fixed to the calibration plate 4. Control structure, the control structure includes a structural body, a display screen 19 set on the structural body, and a plurality of control buttons set on the structural body. Specifically, there are 8 remote control buttons, namely: the first button 11, the second button 12, the third button 13, the fourth button 14, the fifth button 15, the sixth button 16, the seventh button 17 and the eighth button 18. The height of the calibration bracket 5 and the status of the Bluetooth connection can be displayed on the display screen 9. The length of the calibration bracket 5 is 7 meters. The calibration bracket 5 is an electric telescopic rod. The calibration bracket 5 is electrically connected to the control structure. The first button 11 and the fifth button 15 of the calibration bracket 5 are respectively used to control the extension or contraction of the calibration bracket 5. The problem of difficulty in calibrating the height of the on-site camera 7 is solved. The first motor 1 is electrically connected to the control structure. The second button 12 and the sixth button 16 can be used to control the first motor 1 to turn left or right respectively. In any achievable embodiment, the rotation angle of the first motor 1 generated each time is 5 degrees. The second motor 2 is electrically connected to the control structure. The third button 13 and the fourth button 17 can be used to control the second motor 2 to spin up or down respectively. In any achievable embodiment, the rotation angle of the second motor 2 generated each time is 5 degrees. The second motor 2 can be used to adjust the angle of the plane of the calibration plate 4 relative to the on-site camera 7. The third motor 3 is electrically connected to the control structure. The fourth button 14 and the eighth button 18 can be used to control the left or right rotation of the third motor 3, respectively. In any feasible embodiment, the rotation angle of the third motor 3 is 5 degrees each time. The third motor 3 can be used to adjust the horizontal, vertical, left-leaning and right-leaning postures of the calibration plate 4. The walking structure 6 is a universal wheel structure, which is arranged at the bottom end of the calibration bracket 5 and can be used for the movement of the device. The material of the device is mainly made of carbon fiber material, which is light and convenient, and one person can solve the calibration task. The use of the first calibration component effectively solves the problem that in the prior art, in actual camera calibration, the cameras are already installed on the wall and other positions at different heights. If only the calibration plate is used for calibration, it is suitable for the camera to leave the factory or at a lower height.If a handheld calibration rod is used to clamp the calibration plate for calibration, the hand-held calibration rod will shake violently, and it is difficult to take a suitable screenshot of the calibration plate. It is tiring to hold the plate for a long time, and multiple people need to take turns to perform the calibration work.

[0047] The second calibration component includes a remote control, a structural support 36, a fourth motor, a fifth motor, a sixth motor, a seventh motor, an eighth motor, a ninth motor calibration cloth 30 and a second walking structure 37. The bottom end of the structural support 36 is connected and fixed to the second walking structure 37. The remote control and the calibration structure are connected via Bluetooth. In the present embodiment, each grid of the calibration cloth is a 50cm*50cm square. The length of the structural support 36 is 2 meters. The structural support 36 is a 2-meter-high telescopic support for the ground. In the present embodiment, the setting of the second walking structure 37 can meet the movement of the calibration device forward, backward, left and right. The material of this set of equipment is mainly made of carbon fiber material, which is light and convenient, and one person can solve the calibration task.

[0048] In this embodiment, the remote controller includes a structural body, a second display screen 29 disposed on the structural body, and a plurality of control buttons disposed on the structural body. Specifically, there are 8 remote control buttons, namely: a first button 21, a second button 22, a third button 23, a fourth button 24, a fifth button 25, a sixth button 26, a seventh button 27, and an eighth button 28. The second display screen 29 displays the current height of the bracket, the status of the Bluetooth connection, and the current layout of the structure.

[0049] The first button 21 and the second button 22 are used to control the lifting of the structural support 36, which is used to solve the calibration problem of different heights. The third button 23 is used to control the fourth motor, and the eighth button 28 is a confirmation button. The third button 23 is used to control the fourth motor. Clicking the eighth button 28 will expand the top of the structural support 36. Figure 5-1 The fourth button 24 is used to control the fifth motor. The fourth button 24 is used to control the fifth motor. Click the eighth button 28 to expand the top of the structural support 36 as shown in FIG. Figure 5-2 The fifth button 25 is used to control the sixth motor. The fifth button 25 is used to control the sixth motor. Click the eighth button 28 to expand the top of the structural support 36 as shown in FIG. Figure 5-3 The sixth button 26 is used to control the seventh motor and the eighth motor. The sixth button 26 is used to control the seventh motor and the eighth motor. Click the eighth button 28 to expand the top of the structural support 36. Figure 5-4 The seventh button 27 is used to control the ninth motor. The seventh button 27 is used to control the ninth motor. Click the eighth button 28 to expand the top of the structural support 36. Figure 5-5 layout.

[0050] The use of the second calibration component effectively solves the problem that most camera calibrations in the prior art are performed on the ground using a calibration cloth. The final coordinate conversion also uses the ground two-dimensional coordinate system, which is severely blocked by high-density pedestrian footsteps and cannot be restored by identifying the foot position; the calibration at a certain height uses four tripods, which are adjusted to the same height, and the distance between the racks must be measured to form a standard square. The operation is relatively complicated, and the four tripods are also relatively heavy. At least two people are required to cooperate in the calibration process to complete the calibration.

[0051] S2. Use the parameter calibration system in combination with the screenshot to calibrate the coordinate transformation model and area parameters, and save the calibration parameters and the corresponding camera number; that is, calibrate the camera intrinsic parameters and scene extrinsic parameters through the height-adjustable calibration device.

[0052] Preferably, step S2 comprises the following steps:

[0053] S2.1. Calculate the internal parameters of the calibration plate. Specifically, capture multiple sets of calibration plate images at different positions and angles under the camera; identify the corner points of the calibration plate in the image; calculate the camera internal parameter matrix and distortion coefficient; and generate a correction image and new internal parameters after correction.

[0054] In this step, a calibration operation is performed using the first calibration component.

[0055] The implementation of this step includes: obtaining the transformation relationship from the space coordinate system to the camera coordinate system.

[0056] Assume that the coordinates of the corner points of the calibration plate in the spatial coordinate system are (X a ,Y a ,Z a ) (Usually the calibration plate plane is placed in the spatial coordinate system Z a =0), the coordinates in the camera coordinate system are (X b ,Y b ,Z b ), the transformation relationship from the space coordinate system to the camera coordinate system is represented by the rotation matrix R and the translation vector t, that is,

[0057] Solve for each parameter in the internal parameter matrix.

[0058] Assume the coordinates in the pixel coordinate system are (u, v), and introduce the focal length f χ 、f y , image center coordinates (u0, v0) and pixel offset coefficient c, the conversion relationship is where s χ and y is the number of pixels per unit length. The least squares method and other optimization algorithms are used to solve the internal parameter matrix The various parameters in .

[0059] In this step, the process of generating the correction map and the new internal parameters after correction is as follows:

[0060] Calculate the distortion coefficient and perform distortion correction on the image;

[0061] The tangential distortion formula is:

[0062] x 畸变 =x+(2*p1*x*y+p2*(r 2 +2*x 2 )),y 畸变 =y+(p1*(r 2 +2*y 2 )+2*p2*x*y), abnormal

[0063] Variable correction formula:

[0064] x=x 畸变 -2p1x 畸变 y 畸变 +p2r 2 +2x 畸变 2 ,

[0065] Where p1, p2 are tangential distortion coefficients; r 2 =x 2 +y 2 . x, y are the coordinates of the pixel points in the image without considering the distortion. 畸变 ,y 畸变 is the pixel coordinate after distortion, p1, p2 are the tangential distortion coefficients, which are obtained through camera calibration and describe the degree of tangential distortion. r represents the distance from the pixel x, y to the center of the image, and r 2 =x 2 +y 2 ,

[0066] The radial distortion formula is:

[0067] x 畸变 =x(1+k1*r 2 +k2*r 4 +k3*r 6 ), y 畸变 =y(1+k1*r 2 +k2*r 4 +k3*r 6 ), distortion correction: given the distortion coefficient, for any point (x, y) in the image, the distortion correction is performed by the inverse operation of the above distortion formula, and the corrected coordinates are: Where x and y are the coordinates of the pixel points in the image without considering the distortion. 畸变 ,y 畸变 is the pixel coordinate after distortion, k1, k2, k3 are radial distortion coefficients, k1 coefficient is mainly used to describe first-order radial distortion, if k1>0 lens usually shows pincushion distortion, if k1<0 lens usually shows barrel distortion, k2, k3 are high-order distortion coefficients, used to more accurately describe complex distortion, used for distortion correction. r represents the distance from pixel x, y to the center of the image, and r 2 =x 2 +y 2 ,

[0068] After distortion correction, obtain the new internal reference A' in the same way as the internal reference

[0069] S2.2. Perform scene calibration, use the calibration cloth to capture pictures according to the average height, and then mark multiple pixel coordinate reference points in the screenshots. Specifically, the average height includes: the average height of children, the average height of teenagers, and the average height of adults. The number of pixel coordinate reference points is not less than 4.

[0070] In any achievable embodiment, the average height of children is 1 meter, the average height of teenagers is 1.2 meters, and the average height of adults is 1.5 meters.

[0071] S2.3. Generate a two-dimensional world coordinate system according to the actual scene; identify the world coordinate system corresponding to the corner point coordinates of the calibration cloth in the coordinate system; and store the world coordinates of the calibration cloth.

[0072] That is, a two-dimensional pseudo-world coordinate system is generated, and the positions of the reference points of the calibration cloth are marked and stored.

[0073] S2.4, solve the extrinsic parameter conversion parameters. Specifically, based on the known camera intrinsic parameters and distortion coefficients; the known calibration cloth pixel coordinates and world coordinates; solve the camera extrinsic parameter rotation vector and translation vector; solve the homography matrix from the correction image to the world coordinates; the correction image coordinates are transformed to the world coordinates using the homography.

[0074] More specifically, the camera extrinsic rotation vector and translation vector are solved using the pnp method;

[0075] In this step, the process of solving the camera extrinsic rotation vector and translation vector includes:

[0076] Select four 3D points in the world coordinate system (X a ,Y a ,Z a )(X b ,Y b ,Zb )(X c ,Y c ,Z c )(X d ,Y d ,Z d ) and the 2D points (u' a ,v' a ,1' a )(u' b ,v' b ,1' b )(u' c ,v' c ,1' c )(u' d ,v' d ,1' d ) corresponds to, assuming that the 3D point P a =(X a ,Y a ,Z a ) The coordinates of the corresponding 2D point in the pixel coordinate system are m a =(u a ,v a ,1 a ). Known camera intrinsic parameters According to the perspective projection formula: s is a scaling factor used to balance the loss of depth information during the planar projection process from 3D space to 2D images. is the pixel coordinate of the 2D point image, u is the horizontal coordinate, and v is the vertical coordinate. A is the intrinsic parameter matrix calibrated by the camera P a is the coordinate of the 3D point, R is the rotation matrix, t=(t x ,t y ,t x ) T For the translation vector, the final camera extrinsic parameters are calculated by establishing the projection relationship equation and selecting an appropriate solution method (such as direct linear transformation DLT or iterative optimization method).

[0077] According to the new intrinsic and extrinsic parameters of the camera, the calibration cloth is converted to the coordinates of the calibration cloth after pseudo correction. Given the corrected coordinates of the calibration cloth and the corresponding two-dimensional coordinates of the pseudo world, the homography matrix from the coordinates of the correction image to the coordinates of the pseudo world is calculated. The formula of the homography matrix is: The 2D pixel coordinate u is the horizontal coordinate, and v is the vertical coordinate corresponding to the X and Y coordinates of the 3D point. The inverse solution is used to obtain the homography matrix H. The matrix is ​​a 3x3 matrix that contains the set transformation information from the source point coordinates to the target coordinates. The element h 11 and h 12 Related to scaling and rotation, h13 and h 23 Mainly involves the translation part.

[0078] The correction image can be transformed into the pseudo-world two-dimensional coordinates using the homography matrix, that is, the pixel coordinates of the head can be transformed into the two-dimensional space coordinates of the head after correction.

[0079] Coordinate transformation formula:

[0080] u i =h 11 x i +h 12 y i +h 13 , v i =h 21 x i +h 22 y i +h 23 ,ω i =h 31 x i +h 32 y i +h 33 . The element h 11 and h 12 Related to scaling and rotation, h 13 and h 23 Mainly involves the translation part, x i ,y i is the source coordinate point, u i is the horizontal coordinate after transformation, v i is the converted vertical coordinate.

[0081] The proportional relationship is calculated based on the actual distance of the calibration points and the pixel distance of the calibration points in the two-dimensional space coordinate system. The two-dimensional space coordinate pixel represents the actual number of centimeters.

[0082] The density area, speed area, queuing area, and out-of-bounds area are marked in the correction image and converted to the two-dimensional space coordinate system of the head through conversion parameters. According to the proportional relationship, the area parameters of the density area, the total length parameters of the queuing area, and the conversion parameters of the pixel distance and actual distance of the speed area are calculated.

[0083] S3. Read the camera configuration file to be detected. The configuration file includes the code stream address, camera code, pre-allocated GPU index, and loads the coordinate conversion model and regional parameters according to the camera number. Read the camera code stream, perform hardware decoding on the video code stream, and extract frames to store the queue. Read the video frames in the queue, generate pedestrian IDs for pedestrian detection and tracking, bind the head pixel coordinate information and age information to the ID; identify the age information of the person according to the detection model. That is, obtain the conversion model of the head pixel coordinates of each age group to the head two-dimensional space coordinate system. Then calibrate the regional parameters through the conversion model, and the regional parameters are based on the two-dimensional space coordinate system of the adult head.

[0084] This step is the camera decoding frame extraction inference stage.

[0085] According to the number of cameras to be detected, the corresponding process is assigned to read the corresponding configuration file.

[0086] The configuration file includes GPU index, camera number, stream address, detection model address, coordinate conversion model parameters and area parameters.

[0087] Call the GPU according to the GPU index to hard-decode the video stream and store the queue, then load the inference network model, load the coordinate transformation model and area parameters.

[0088] The implementation of this step may include the following existing technologies. For example, the two-dimensional coordinate information of the human head space is detected through an inference model. The multi-target tracking algorithm uses a deepsort algorithm. The detected target is re-identified and the target's features are extracted. The Hungarian algorithm is used to associate the tracking box tracks in the previous frame with the detection box detections in the current frame. The cost matrix is ​​calculated using appearance information and Mahalanobis distance, or IOU. The position of the track at the next moment is predicted by Kalman filtering, the predicted position is updated based on detection, and the ID of the detected person is output.

[0089] S4. According to the different age information of pedestrians, the head pixel coordinate information is brought into the coordinate conversion model at different heights, and the actual spatial position is calculated. That is, the age group of the person is identified, and the person is brought into different head two-dimensional spatial coordinates to calculate his actual position;

[0090] In this step, the calibration system calibrates different heights. The homography matrix parameters of the pixel coordinates of 1 meter for children, 1.2 meters for teenagers, and 1.5 meters for adults can be determined to convert the pixel coordinates into two-dimensional world coordinates, and the actual spatial position can be calculated. The implementation of this step includes:

[0091] S4.1. According to the age information of the identified pedestrian, the pixel coordinates of the pedestrian's head are converted to the corrected pixel coordinates of the head through the camera's internal parameters. The camera distortion coefficient is known. For any point (x, y) in the image, the distortion correction is performed through the inverse operation of the above distortion formula. The corrected coordinates are:

[0092] The same is true for tangential distortion.

[0093] S4.2. The corrected head pixel coordinates are brought into the coordinate transformation model of the corresponding age, and transformed into the two-dimensional space coordinate system of the head, that is, the actual space coordinates, according to the homography matrix.

[0094] Conversion formula: u i =h 11 x i +h 12 y i +h 13 , v i =h 21 x i +h 22 y i +h 23 ,ω i =h 31 x i +h 32 y i +h 33

[0095] S5. The positions are uniformly converted into parameter coordinates of the adult area; it is determined whether the corresponding parameter calculation valid area is entered, and the passenger flow parameters are calculated; that is, according to the actual spatial position, the corresponding passenger flow parameters are calculated in the area and the results are output. Among them, the area includes density area, speed area, queuing area, and out-of-bounds area. The above process is implemented using a ray intersection algorithm. Specifically, the point to be judged emits a ray in any direction, and then counts the number of intersections between the ray and the polygon boundary. If the number of intersections is an odd number, the point is inside the polygon; if the number of intersections is an even number, the point is outside the polygon. More specifically, the vertex sequence of the polygon is p={p 0, p 1, p 2, ....p n-1}, p represents the set of vertex sequences of the polygon, p 0, p 1, p 2, ....p n-1 Represent the coordinates of each vertex of the polygon, where p i =(x i, y i ), the point to be judged is m=(x mi, y mi), a ray is emitted from point m to the horizontal direction (the square of the x-axis), and the ray equation is y = y m (Because it is a horizontal ray). For each edge of the polygon pipi +1 (When i=n-1, the next vertex p0 forms a closed polygon), the straight line equation of the edge can be expressed in two-point form: Calculate the intersection point of the ray and the edge, let y = y m , solving for the value of x, we get To determine whether the intersection is valid, the x coordinate of the intersection must be greater than the x coordinate of point m, that is, x>x m . At the same time, the intersection point must be on the edge pipi +1 This can be determined by judging whether the y coordinate of the intersection point is within the y coordinate range of the two end points of the edge, that is, min(y i ,y i+1 )≤y m ≤max(y i ,y i+1 ). Count the number of crossings, traverse all the edges of the polygon, and calculate the validity of the intersection points in the above way. If a valid point is found, the number of crossings increases by 1. If the number of crossings is odd, point m is inside the polygon, and if it is even, it is outside.

[0096] Average density calculation: The coordinate set of the density area and the area S of the density area have been obtained. The passenger flow density parameters are calculated once per second. The density is calculated once every 25 frames. The number of people n whose coordinates are detected and tracked in the current frame are counted in the density area. The passenger flow density is obtained by the ratio of the effective density area S. The formula is: avgd = n / s.

[0097] Heat map calculation: Data point definition, pedestrians enter the density area, the two-dimensional spatial coordinates of the pedestrian's head are used as the x, y coordinates of the data point, the number of each xy is collected per second, and the sum of each xy number in 10 seconds is averaged by 10 to obtain the current average number of each xy, that is, the density value, to complete the data point collection. The actual spacing of 100 cm is converted to the distance between the two-dimensional spatial coordinates of the pedestrian's head according to the proportional relationship to divide the thermal grid, and the actual 30 cm is converted to the distance between the two-dimensional spatial coordinates of the pedestrian's head according to the proportional relationship to set the data point radius. Given the data point position, density value, influence radius and divided grid parameters, the grid heat is further calculated. Traverse the data points to determine their position in the grid. Calculate the heat contribution of each grid unit, and use the Gaussian function to calculate the heat contribution of the data point to the grid unit based on the position and density value of the data point and the distance between the grid unit and the data point. Accumulate the heat contribution of the grid units to obtain the final heat value, and normalize the heat value according to the historical maximum and minimum values. The color gradient setting divides the gradient into 6 levels: 0.1 dark blue, 0.2 light blue, 0.3 green, 0.5 yellow, 0.75 orange, and 1.0 red.

[0098] Speed ​​calculation: real-time speed, calculate personal speed once per second, represented by i in the formula for generating ID for initial tracking. When the 2D head space coordinate of the ID enters the 2D head space coordinate of the speed area, record the 2D pixel coordinate of the head space of the current ID. After every 25 frames, calculate the pixel distance between the latest pixel coordinate of the ID and the pixel coordinate recorded last time. Pixel values, through the proportional relationship (p / d): p is the pixel value in the two-dimensional head space, representing the actual d centimeters, divided by (p / d) to get the actual speed centimeters, and then divided by 100 to get the actual speed of the current ID in meters. The real-time speed calculation formula is: Average speed: add up the speeds of all calculated pedestrians in the speed zone and divide by the number of people in the speed zone n. The formula is: avgs = ∑ism / n. Average linear speed: calculate the pixel distance between the last coordinate of each person appearing in the speed zone and the coordinate of the first entry into the speed zone, convert it to the actual walking distance, and divide it by the difference between the last appearance time and the first appearance time to get the linear speed of each person. where t e represents the first entry time, t s Represents the first entry time. The average linear speed avgxs = ∑ismx / nx is calculated by summing up the linear speeds in the time period and dividing by the number of people, where ismx is the individual linear speed and nx is the number of people passing through.

[0099] Calculate the queue parameters, calibrate the generated queue area, and divide the queue area into single-channel straight-line channels and S-shaped and Z-shaped channels. S-shaped and Z-shaped channels need to be recorded as a collection of multiple straight-line channels. Each straight-line channel is a queue area. The length of the queue area can be calculated through the proportional relationship. The actual queue length will be converted into the actual distance by the pixel difference between the coordinates xa, ya of the first person in the queue and the coordinates xb, yb of the last person in the queue. The actual queue length of the current channel can be obtained as dm = dcm / 100. The actual queue length of the S-shaped channel is obtained by adding up the actual queue lengths of multiple straight lines to obtain the actual queue length of the S-shaped channel sumdm = ∑dm. To calculate the expected queue time t, divide the queue length by the average speed of pedestrians (avgs in speed calculation) t = sumdm / avgs.

[0100] Cross-border calculation: To determine whether a pedestrian has entered the cross-border area, it is necessary to count the pedestrian's entry time. The first time a pedestrian enters, a horn notification warning is called through the serial port protocol. The number of warnings is 3, and the alarm record is retained in the database. If the person leaves, the alarm is eliminated and the alarm record is deleted. If the entry time is as long as 1 minute and there are 6 consecutive warnings, the database alarm record warning is upgraded and the relevant staff is notified to take measures. If the person leaves, the alarm is eliminated and the alarm record is deleted.

[0101] The present invention provides a three-layer network passenger flow parameter calculation method and calibration system. Compared with the prior art, the crowd is divided into three stages according to the size of the crowd's height, namely children, teenagers, and adults, and the average height of each stage is determined respectively. The average height of each stage is used as a two-dimensional space coordinate system (i.e., a two-dimensional space coordinate system of the head), and relevant calibration tools and parameter calibration schemes are customized. The pixel coordinates and the conversion parameters of the head two-dimensional space coordinate system and the regional parameters based on the head two-dimensional space coordinate system are calibrated, including parameters such as effective density area, speed area, cross-border area, and queue area. In target detection, it is first detected whether it is a child, teenager or adult, and then the pixel coordinates of the person's head are converted into the two-dimensional space coordinate system of the head of children, teenagers and adults through the conversion parameters to calculate their actual positions. According to the actual position, the regional parameters are brought in to further calculate multiple passenger flow parameters including personnel density, speed, queue length, duration, whether cross-border abnormal behavior occurs, etc. The calibration of two-dimensional space coordinate systems at different heights is realized. The detection and calculation of real-time passenger flow parameters of the three-layer network is realized.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-layer network passenger flow parameter calculation method, characterized in that: The following steps are involved: Use the calibration device to calibrate the camera to be tested and take a screenshot; Use the parameter calibration system combined with the screenshot to calibrate the coordinate transformation model and area parameters, and save the calibration parameters and the corresponding camera number; Read the configuration file of the camera to be tested, read the camera code stream, perform hardware decoding on the video code stream, and extract frames to store in the queue; Then read the video frames in the queue, detect and track pedestrians to generate pedestrian IDs, and bind the head pixel coordinate information and age information to the IDs; According to the different age information of pedestrians, the head pixel coordinate information is brought into the coordinate conversion model at different heights to calculate the actual spatial position; According to the actual spatial position, the calculation is reversed to the area, the corresponding passenger flow parameters are calculated and the results are output.

2. A three-layer network passenger flow parameter calculation method as claimed in claim 1, characterized in that: The parameter calibration system is used to calibrate the coordinate transformation model and the regional parameters in combination with the screenshot, and the calibration parameters and the corresponding camera number are saved, including: Capture multiple sets of calibration plate images at different positions and angles under the camera; identify the corner points of the calibration plate in the image; calculate the camera intrinsic parameter matrix and distortion coefficient; and generate the correction image and the new internal parameter after correction; Perform scene calibration, use the calibration cloth to capture pictures according to the average height, and then mark multiple pixel coordinate reference points in the screenshots; Generate a two-dimensional world coordinate system according to the actual scene; identify the world coordinate system corresponding to the corner coordinates of the calibration cloth in the coordinate system; store the world coordinates of the calibration cloth; Solve for the external parameters for the transformation.

3. A three-layer network passenger flow parameter calculation method as claimed in claim 2, characterized in that: The step of bringing the head pixel coordinate information into coordinate conversion models at different heights according to the different age information of pedestrians to calculate the actual spatial position includes: According to the age information of the identified pedestrian, the pixel coordinates of the pedestrian's head are converted to the corrected pixel coordinates of the head through the camera's internal parameters. The camera distortion coefficient is known. For any point (x, y) in the image, the corrected coordinates are: The same is true for tangential distortion; k1, k2, k3 are radial distortion coefficients; r represents the distance from the pixel point x, y to the center of the image; The corrected head pixel coordinates are brought into the coordinate transformation model of the corresponding age and transformed into the two-dimensional space coordinate system of the head according to the homography matrix, that is, the actual space coordinates.

4. A three-layer network passenger flow parameter calculation method as claimed in claim 2, characterized in that: The method captures multiple groups of calibration plate images at different positions and angles under the camera; and identifies the corner points of the calibration plate in the images; Calculate the camera internal parameter matrix and distortion coefficients; and generate the correction map and the new internal parameters after correction, including: Assume that the coordinates of the corner points of the calibration plate in the spatial coordinate system are (X a ,Y a ,Z a ), the coordinates in the camera coordinate system are (X b ,Y b ,Z b ), the transformation relationship from the space coordinate system to the camera coordinate system is represented by the rotation matrix R and the translation vector t, that is, 5. A three-layer network passenger flow parameter calculation method as claimed in claim 4, characterized in that: The method captures multiple groups of calibration plate images at different positions and angles under the camera; and identifies the corner points of the calibration plate in the images; Calculate the camera internal parameter matrix and distortion coefficients; and generate the correction map and the new internal parameters after correction, including: The tangential distortion formula is: x 畸变 = x + (2 * p1 * x * y + p2 * (r 2 + 2 * x 2 )),y 畸变 = y + (p1 * (r 2 + 2 * y 2 ) + 2 * p2 * x * y),strange Variable correction formula: x=x 畸变 -2p1x 畸变 y 畸变 +p2r 2 +2x 畸变 2 , Where p1, p2 are tangential distortion coefficients; r 2 =x 2 +y 2 ; x, y are the coordinates of the pixel points in the image without considering the distortion, x 畸变 ,y 畸变 is the pixel coordinate after distortion, p1, p2 are the tangential distortion coefficients, which are obtained through camera calibration and describe the degree of tangential distortion; r represents the distance from the pixel x, y to the center of the image, and r 2 =x 2 +y 2 , The radial distortion formula is: x 畸变 =x(1+k1*r 2 +k2*r 4 +k3*r 6 ),y 畸变 =y(1+k1*r 2 +k2*r 4 +k3*r 6 ); Distortion correction: Given a known distortion coefficient, for any point (x, y) in the image, distortion correction is performed by inverse calculation of the above distortion formula. The corrected coordinates are: Among them, x, y are the coordinates of the pixel points in the image without considering the distortion, x 畸变 ,y 畸变 is the pixel coordinate after distortion, k1, k2, k3 are radial distortion coefficients, k1 coefficient is mainly used to describe first-order radial distortion, if k1>0 lens usually shows pincushion distortion, if k1<0 lens usually shows barrel distortion, k2, k3 are high-order distortion coefficients, used to more accurately describe complex distortion, used for distortion correction; r represents the distance from pixel x, y to the center of the image, and r 2 =x 2 +y 2 , 6. A three-layer network passenger flow parameter calculation method as claimed in claim 4, characterized in that: The method captures multiple groups of calibration plate images at different positions and angles under the camera; and identifies the corner points of the calibration plate in the images; Calculate the camera internal parameter matrix and distortion coefficients; and generate the correction map and the new internal parameters after correction, including: Assume the coordinates in the pixel coordinate system are (u, v), and introduce the focal length f χ 、f y , image center coordinates (u0, v0) and pixel offset coefficient c, the conversion relationship is where s χ and y The number of pixels per unit length.

7. A three-layer network passenger flow parameter calculation method as claimed in claim 4, characterized in that: The method captures multiple groups of calibration plate images at different positions and angles under the camera; and identifies the corner points of the calibration plate in the images; Calculate the camera internal parameter matrix and distortion coefficients; generate the correction map and the new internal parameters after correction, including: using the least squares optimization algorithm to solve the internal parameter matrix The various parameters in .

8. A three-layer network passenger flow parameter calculation method as claimed in claim 1, characterized in that: The method of back-calculating into an area based on the actual spatial position, calculating corresponding passenger flow parameters and outputting the results includes: the area includes one or more of a density area, a speed area, a queuing area, and an out-of-bounds area.

9. A three-layer network passenger flow parameter calculation method as claimed in claim 1, characterized in that: The configuration file includes: one or more of a gpu index, a camera number, a code stream address, a detection model address, a coordinate conversion model parameter, and an area parameter.

10. A calibration system for calculating passenger flow parameters of a three-layer network, characterized in that: Used in the steps of a three-layer network passenger flow parameter calculation method as claimed in any one of claims 1 to 9, comprising: a first calibration component and a second calibration component; The first calibration component includes a control structure, a calibration bracket, a first motor, a second motor, a third motor, a calibration board, and a walking structure; the bottom end of the calibration bracket is connected and fixed to the walking structure; the top end of the calibration bracket is connected and fixed with the first motor; the first motor is connected to the second motor through a first transmission rod; the second motor is connected to the third motor through a second transmission rod; the third motor is connected and fixed to the calibration board; The second calibration component includes a remote control, a structural support, a fourth motor, a fifth motor, a sixth motor, a seventh motor, an eighth motor, a ninth motor calibration cloth and a second walking structure.

Images