A three-layer network passenger flow parameter calculation method and calibration system

Abstract

The present invention provides a three-layer network passenger flow parameter calculation method and calibration system, comprising: using a calibration device to calibrate the camera to be detected and take a screenshot; using a parameter calibration system in combination with the screenshot to determine a coordinate conversion model and regional parameters, and saving the calibration parameters and the corresponding camera number; reading the configuration file of the camera to be detected, reading the camera code stream, performing hardware decoding on the video code stream, extracting frames and storing them in a queue; then reading the video frames in the queue, detecting and tracking pedestrians to generate pedestrian IDs, and binding head pixel coordinate information and age information to the IDs; according to the different age information of the pedestrians, the head pixel coordinate information is brought into a coordinate conversion model at different heights to calculate the actual spatial position; based on the actual spatial position, the actual spatial position is reversely calculated into the region, the corresponding passenger flow parameters are calculated, and the results are output. The method and system provided by the present invention realize the calibration of two-dimensional spatial coordinate systems at different heights and the detection and calculation of real-time passenger flow parameters.

Description

Technical Field

[0001] The present invention belongs to the field of rail transportation technology, and more specifically, relates to a three-layer network passenger flow parameter calculation method and calibration system. Background Art

[0002] Many cities in my country have subways with large passenger flows. To address the high passenger flow caused by transportation, tourist attractions, and large-scale events, video detection technology is often used to monitor passenger flow.

[0003] Existing video detection technology typically uses ground-based reference objects to calibrate the ground's two-dimensional coordinate system, then captures camera images to calculate intrinsic parameters. This existing technology suffers from the fact that most cameras in real-world scenarios are fixed-angle, lacking multi-angle and multi-position camera images, leading to significant errors in the calculation results. Summary of the Invention

[0004] The purpose of the present invention is to provide a three-layer network passenger flow parameter calculation method and calibration system, aiming to solve the technical problem that the calculation results in the prior art have large errors.

[0005] To achieve the above object, the present invention adopts a technical solution of providing a three-layer network passenger flow parameter calculation method, comprising the following steps:

[0006] Use the calibration device to calibrate the camera to be tested and take a screenshot;

[0007] Use the parameter calibration system combined with the screenshot to calibrate the coordinate transformation model and area parameters, and save the calibration parameters and the corresponding camera number;

[0008] Read the configuration file of the camera to be detected, read the camera code stream, perform hardware decoding on the video code stream, extract frames and store them in a queue; then read the video frames in the queue, detect and track pedestrians, generate pedestrian IDs, and bind the head pixel coordinate information and age information to the ID;

[0009] Based on the pedestrian's age information, the head pixel coordinate information is brought into the coordinate transformation model at different heights to calculate the actual spatial position;

[0010] According to the actual spatial position, reverse calculation is performed to the area, the corresponding passenger flow parameters are calculated and the results are output.

[0011] Preferably, the parameter calibration system is used in combination with the screenshot to calibrate the coordinate transformation model and the area parameters, and the calibration parameters and the corresponding camera number are saved, including:

[0012] Capture multiple sets of calibration plate images at different positions and angles under the camera; identify the corner points of the calibration plate in the image; calculate the camera intrinsic parameter matrix and distortion coefficient; and generate the correction image and the new internal parameters after correction;

[0013] Perform scene calibration by taking screenshots at the average height using a calibration cloth, and then marking multiple pixel coordinate reference points in the screenshots.

[0014] Generate a two-dimensional world coordinate system according to the actual scene; identify the world coordinate system corresponding to the corner coordinates of the calibration cloth in the coordinate system; store the world coordinates of the calibration cloth;

[0015] Solve for the external parameters for the transformation.

[0016] Preferably, the step of bringing the head pixel coordinate information into coordinate conversion models at different heights according to the different age information of the pedestrians to calculate the actual spatial position includes:

[0017] According to the age information of the identified pedestrian, the pixel coordinates of the pedestrian's head are converted to the corrected head pixel coordinates using the camera's intrinsic parameters. Given the camera distortion coefficient, for any point (x, y) in the image, the distortion correction is performed by inverse operation of the above distortion formula. The corrected coordinates are:

[0018] The same applies to tangential distortion;

[0019] The corrected head pixel coordinates are brought into the coordinate transformation model of the corresponding age and transformed into the head two-dimensional space coordinate system, i.e., the actual space coordinates, according to the homography matrix. The transformation formula is:

[0020] u i =h 11 x i +h 12 y i +h 13 , v i =h 21 x i +h 22 y i +h 23 ,ω i =h 31 x i +h 32 y i +h 33 .

[0021] Preferably, the method of capturing multiple sets of calibration plate images at different positions and angles under the camera; identifying the corner points of the calibration plate in the images; calculating the camera intrinsic parameter matrix and distortion coefficient; and generating the correction image and the corrected new intrinsic parameters includes:

[0022] Assume that the coordinates of the corner points of the calibration plate in the space coordinate system are (X a ,Y a ,Z a ) (Usually the calibration plate plane is placed in the spatial coordinate system Z a =0), the coordinates in the camera coordinate system are (X b ,Y b ,Z b ), the transformation relationship from the space coordinate system to the camera coordinate system is represented by the rotation matrix R and the translation vector t, that is,

[0023] Preferably, the method of capturing multiple sets of calibration plate images at different positions and angles under the camera; identifying the corner points of the calibration plate in the images; calculating the camera intrinsic parameter matrix and distortion coefficient; and generating the correction image and the corrected new intrinsic parameters includes:

[0024] The tangential distortion formula is:

[0025] x 畸变 =x+(2*p1*x*y+p2*(r 2 +2*x 2 )),y 畸变 =y+(p1*(r 2 +2*y 2 )+2*p2*x*y) where p1 and p2 are tangential distortion coefficients; r 2 =x 2 +y 2 .

[0026] The radial distortion formula is:

[0027] x 畸变 =x(1+k1*r 2 +k2*r 4 +k3*r 6 ), y 畸变 =y(1+k1*r 2 +k2*r 4 +k3*r 6 ), where k1, k2, and k3 are radial distortion coefficients.

[0028] Distortion correction: Given a known distortion coefficient, for any point (x, y) in the image, distortion correction is performed by inverse operation of the above distortion formula. The corrected coordinates are:

[0029]

[0030] Preferably, the method of capturing multiple sets of calibration plate images at different positions and angles under the camera; identifying the corner points of the calibration plate in the images; calculating the camera intrinsic parameter matrix and distortion coefficient; and generating the correction image and the corrected new intrinsic parameters includes:

[0031] Assume the coordinates in the pixel coordinate system are (u, v), and introduce the focal length f χ 、f y , image center coordinates (u0, v0) and pixel offset coefficient c, the conversion relationship is where s χ and s y The number of pixels per unit length.

[0032] Preferably, the method of capturing multiple sets of calibration plate images at different positions and angles under the camera; identifying the corner points of the calibration plate in the image; calculating the camera intrinsic parameter matrix and distortion coefficient; and generating the correction image and the new internal parameter after correction includes: using an optimization algorithm such as the least squares method to solve the intrinsic parameter matrix The various parameters in .

[0033] Preferably, the back-calculation into the area based on the actual spatial position, calculating the corresponding passenger flow parameters and outputting the results includes: the area includes one or more of a density area, a speed area, a queuing area, and an out-of-bounds area.

[0034] Preferably, the configuration file includes one or more of: a GPU index, a camera number, a code stream address, a detection model address, coordinate conversion model parameters, and area parameters.

[0035] The present invention also provides a calibration system for calculating passenger flow parameters of a three-layer network, characterized in that it is used in the steps of a three-layer network passenger flow parameter calculation method as described in any one of the above items, comprising: a first calibration component and a second calibration component;

[0036] The first calibration assembly includes a control structure, a calibration bracket, a first motor, a second motor, a third motor, a calibration plate, and a walking structure; the bottom end of the calibration bracket is connected and fixed to the walking structure; the top end of the calibration bracket is connected and fixed to the first motor; the first motor is connected to the second motor via a first transmission rod; the second motor is connected to the third motor via a second transmission rod; and the third motor is connected and fixed to the calibration plate;

[0037] The second calibration component includes a remote control, a structural support, a fourth motor, a fifth motor, a sixth motor, a seventh motor, an eighth motor, a ninth motor calibration cloth and a second walking structure.

[0038] The beneficial effects of the three-layer network passenger flow parameter calculation method and calibration system provided by the present invention are as follows: compared with the existing technology, the three-layer network passenger flow parameter calculation method and calibration system provided by the present invention, in the calibration process, the average height of each stage is divided into three dimensions: the average height of children, teenagers, and adults. The average height is used as a two-dimensional space coordinate system, that is, a two-dimensional space coordinate system of the head. Relevant calibration tools and parameter calibration schemes are customized to calibrate the conversion parameters of the pixel coordinates and the two-dimensional space coordinate system of the head and the regional parameters based on the two-dimensional space coordinate system of the head, including parameters such as the effective density area, speed area, out-of-bounds area, and queue area. In target detection, it is first detected whether it is a child, teenager or adult, and then the pixel coordinates of the person's head are converted into the two-dimensional space coordinate system of the head of children, teenagers and adults through the conversion parameters to calculate their actual position. According to the actual position, the regional parameters are brought in to further calculate multiple passenger flow parameters including personnel density, speed, queue length, duration, whether out-of-bounds abnormal behavior occurs, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of a method for calculating passenger flow parameters in a three-layer network provided by an embodiment of the present invention; Figure 2 A flowchart of step S1 in a three-layer network passenger flow parameter calculation method provided by an embodiment of the present invention; Figure 3 A schematic structural diagram of a first calibration component used in an embodiment of the present invention; Figure 4 A schematic structural diagram of a second calibration component used in an embodiment of the present invention; Figure 5 Schematic diagram of different usage states of the calibration cloth of the second calibration assembly used in an embodiment of the present invention; Figure 6 A schematic diagram of a three-layer network conversion process in a three-layer network passenger flow parameter calculation method provided by an embodiment of the present invention; Figure 7 It is the heat map of the two-dimensional spatial coordinates of the pedestrian's head; Figure 8 Calculate the flow chart for pedestrian speed; Figure 9 Flowchart for pedestrian crossing alarm. DETAILED DESCRIPTION

[0041] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] Existing calibration methods are usually based on the ground coordinate system. The calculation of people's position and speed is based on the conversion of the pixel coordinates of the soles of the feet into the ground two-dimensional space coordinate system. This method has no problem with calculation accuracy when there are few people. However, in reality, many scenes are densely populated, and the feet of most people are blocked in the camera image, making it impossible to calculate people's position and walking speed by identifying their feet.

[0043] The present invention provides a three-layer network passenger flow parameter calculation method. In order to solve the problem that there are large errors in the calculation of passenger flow parameters in the monitoring screen of dense crowds in the existing technology, a method for calculating passenger flow parameters in dense crowds is provided. During the calibration process, the average height of each stage is divided into three dimensions: the average height of children, teenagers, and adults. The average height is used as a two-dimensional space coordinate system, that is, a two-dimensional space coordinate system of the head. Relevant calibration tools and parameter calibration schemes are customized to calibrate the conversion parameters of pixel coordinates and the two-dimensional space coordinate system of the head and the regional parameters based on the two-dimensional space coordinate system of the head, including parameters such as effective density area, speed area, out-of-bounds area, and queue area. In target detection, it is first detected whether it is a child, teenager or adult, and then the pixel coordinates of the person's head are converted into the two-dimensional space coordinate system of the head of children, teenagers and adults through the conversion parameters to calculate their actual position. According to the actual position, the regional parameters are brought in to further calculate multiple passenger flow parameters including personnel density, speed, queue length, duration, whether out-of-bounds abnormal behavior occurs, etc. For details, please refer to Figures 1 to 9 The present invention provides a method for calculating passenger flow parameters in a three-layer network. The method comprises the following steps:

[0044] S1. Use the calibration device to calibrate the camera to be tested and take a screenshot;

[0045] Preferably, step S1 includes the following steps: S1.1, assembling a calibration device; the calibration device includes a first calibration component and a second calibration component.

[0046] The first calibration component includes a calibration bracket, a calibration plate mounted on the bracket, and a drive motor. The height of the calibration bracket and the angle of the calibration plate can be adjusted via Bluetooth remote control to meet the needs of capturing multiple sets of calibration plate images at different positions and angles. The calibration bracket is a 7-meter-high, floor-standing telescopic bracket. This device consists of two parts: a remote control and a calibration device. The left remote control and the right calibration device are connected via Bluetooth. The first calibration component includes a control structure, a calibration bracket 5, a first motor 1, a second motor 2, a third motor 3, a calibration plate 4, and a walking structure 6. The bottom end of the calibration bracket 5 is connected and fixed to the walking structure 6. The top end of the calibration bracket 5 is connected and fixed to the first motor 1. The first motor 1 is connected to the second motor 2 via a first transmission rod. The second motor 2 is connected to the third motor 3 via a second transmission rod. The third motor 3 is connected and fixed to the calibration plate 4. The control structure includes a structural body, a display screen 19 mounted on the structural body, and multiple control buttons mounted on the structural body. Specifically, the remote control has eight buttons: a first button 11, a second button 12, a third button 13, a fourth button 14, a fifth button 15, a sixth button 16, a seventh button 17, and an eighth button 18. The display screen 9 shows the height of the calibration bracket 5 and the status of the Bluetooth connection. The calibration bracket 5 is 7 meters long and is an electrically retractable rod. It is electrically connected to the control structure. The first button 11 and the fifth button 15 of the calibration bracket 5 are used to control the extension or retraction of the calibration bracket 5, respectively. This solves the problem of difficulty in calibrating the height of the on-site camera 7. The first motor 1 is electrically connected to the control structure. The second button 12 and the sixth button 16 can be used to control the first motor 1 to rotate left or right, respectively. In any feasible embodiment, the first motor 1 rotates by an angle of 5 degrees each time. The second motor 2 is electrically connected to the control structure. The third button 13 and the fourth button 17 can be used to control the second motor 2 to rotate upward or downward, respectively. In any feasible embodiment, the second motor 2 rotates by an angle of 5 degrees each time. The second motor 2 can be used to adjust the angle of the plane of the calibration plate 4 relative to the on-site camera 7. The third motor 3 is electrically connected to the control structure. The fourth button 14 and the eighth button 18 can be used to control the left or right rotation of the third motor 3 respectively. In any feasible embodiment, the rotation angle of the third motor 3 each time is 5 degrees. The third motor 3 can be used to adjust the horizontal, vertical, left and right postures of the calibration plate 4. The walking structure 6 is a universal wheel structure, which is arranged at the bottom end of the calibration bracket 5 and can be used for the movement of the device. The material of the device is mainly carbon fiber material, which is light and convenient, and one person can solve the calibration task. The use of the first calibration component effectively solves the problem of the existing technology that in actual camera calibration, the cameras are already installed on the wall and other locations at different heights. If only the calibration plate is used for calibration, it is suitable for the camera to be shipped from the factory or at a lower height.If a handheld calibration rod is used to clamp the calibration plate for calibration, the hand-held calibration rod will shake greatly, it will be difficult to take a suitable screenshot of the calibration plate, it will be tiring to hold the plate for a long time, and multiple people will need to be replaced to perform the calibration work.

[0047] The second calibration component includes a remote control, a structural support 36, a fourth motor, a fifth motor, a sixth motor, a seventh motor, an eighth motor, a ninth motor calibration cloth 30 and a second walking structure 37. The bottom end of the structural support 36 is connected and fixed to the second walking structure 37. The remote control and the calibration structure are connected via Bluetooth. In this embodiment, each grid of the calibration cloth is a 50cm*50cm square. The length of the structural support 36 is 2 meters. The structural support 36 is a 2-meter-high telescopic support that is ground-mounted. In this embodiment, the setting of the second walking structure 37 can meet the movement of the calibration device forward, backward, left and right. The material of this set of equipment is mainly carbon fiber material, which is light and convenient, and one person can solve the calibration task.

[0048] In this embodiment, the remote control comprises a main structure, a second display screen 29 disposed on the main structure, and a plurality of control buttons disposed on the main structure. Specifically, there are eight remote control buttons: a first button 21, a second button 22, a third button 23, a fourth button 24, a fifth button 25, a sixth button 26, a seventh button 27, and an eighth button 28. The second display screen 29 displays the current stand height, the Bluetooth connection status, and the currently deployed layout of the structure.

[0049] The first button 21 and the second button 22 are used to control the lifting of the structural support 36, which is used to solve the calibration problem of different heights. The third button 23 is used to control the fourth motor, and the eighth button 28 is the confirmation button. The third button 23 is used to control the fourth motor. Clicking the eighth button 28 will expand the top of the structural support 36. Figure 5-1 The fourth button 24 is used to control the fifth motor. The fourth button 24 is used to control the fifth motor. Click the eighth button 28 to expand the top of the structural support 36. Figure 5-2 The fifth button 25 is used to control the sixth motor. The fifth button 25 is used to control the sixth motor. Click the eighth button 28 to expand the top of the structural support 36. Figure 5-3 The sixth button 26 is used to control the seventh motor and the eighth motor. Click the eighth button 28 to expand the top of the structural support 36. Figure 5-4 The seventh button 27 is used to control the ninth motor. The seventh button 27 is used to control the ninth motor. Click the eighth button 28 to expand the top of the structural support 36. Figure 5-5 layout.

[0050] The use of the second calibration component effectively solves the existing problem that most camera calibrations are performed on the ground using a calibration cloth. The final coordinate conversion also uses a ground-based 2D coordinate system, which is severely obscured by dense pedestrian footsteps, making it impossible to recover the actual ground 2D coordinates by identifying foot positions. Calibration at a specific height requires four tripods, all adjusted to the same height and with the distances between them measured to form a standard square. This is a complex operation, and the four tripods are also relatively heavy, requiring at least two people to complete the calibration process.

[0051] S2. Use the parameter calibration system in combination with the screenshot to calibrate the coordinate transformation model and area parameters, and save the calibration parameters and the corresponding camera number; that is, calibrate the camera intrinsic parameters and scene extrinsic parameters through the height-adjustable calibration device.

[0052] Preferably, step S2 includes the following steps:

[0053] S2.1. Determine the intrinsic parameters of the calibration plate. Specifically, capture multiple sets of calibration plate images at different positions and angles under the camera. Identify the corner points of the calibration plate in the image. Calculate the camera's intrinsic parameter matrix and distortion coefficients. Finally, generate a correction image and new corrected internal parameters.

[0054] In this step, a calibration operation is performed using the first calibration component.

[0055] The implementation of this step includes: obtaining the transformation relationship from the space coordinate system to the camera coordinate system.

[0056] Assume that the coordinates of the corner points of the calibration plate in the space coordinate system are (X a ,Y a ,Z a ) (Usually the calibration plate plane is placed in the spatial coordinate system Z a =0), the coordinates in the camera coordinate system are (X b ,Y b ,Z b ), the transformation relationship from the space coordinate system to the camera coordinate system is represented by the rotation matrix R and the translation vector t, that is,

[0057] Solve for each parameter in the internal parameter matrix.

[0058] Assume the coordinates in the pixel coordinate system are (u, v), and introduce the focal length f χ 、f y , image center coordinates (u0, v0) and pixel offset coefficient c, the conversion relationship is where s χ and s y is the number of pixels per unit length. Least squares method and other optimization algorithms are used to solve the internal parameter matrix The various parameters in .

[0059] In this step, the process of generating the correction map and the new internal reference after correction is as follows:

[0060] Calculate the distortion coefficient and perform distortion correction on the image;

[0061] The tangential distortion formula is:

[0062] x 畸变 =x+(2*p1*x*y+p2*(r 2 +2*x 2 )),y 畸变 =y+(p1*(r 2 +2*y 2 )+2*p2*x*y), abnormal

[0063] Variation correction formula:

[0064] x=x 畸变 -2p1x 畸变 y 畸变 +p2r 2 +2x 畸变 2 ,

[0065] Where p1 and p2 are tangential distortion coefficients; r 2 =x 2 +y 2 x,y are the coordinates of the pixel points in the image without considering the distortion, x 畸变 ,y 畸变 is the pixel coordinate after distortion, p1, p2 are the tangential distortion coefficients, which are obtained through camera calibration and describe the degree of tangential distortion. r represents the distance from the pixel x, y to the center of the image, and r 2 =x 2 +y 2 ,

[0066] The radial distortion formula is:

[0067] x 畸变 =x(1+k1*r 2 +k2*r 4 +k3*r 6 ), y 畸变 =y(1+k1*r 2 +k2*r 4 +k3*r 6 ), distortion correction: given the known distortion coefficient, for any point (x, y) in the image, the distortion correction is performed by the inverse operation of the above distortion formula. The corrected coordinates are: Where x, y are the coordinates of the pixel in the image without considering the distortion, x 畸变 ,y 畸变 is the pixel coordinate after distortion, k1, k2, k3 are radial distortion coefficients, k1 coefficient is mainly used to describe the first-order radial distortion, if k1>0 the lens usually shows pincushion distortion, if k1<0 the lens usually shows barrel distortion, k2, k3 are high-order distortion coefficients, used to more accurately describe complex distortion conditions, used for distortion correction. r represents the distance from the pixel x, y to the center of the image, and r 2 =x 2 +y 2 ,

[0068] After distortion correction, obtain the new internal reference A' in the same way as the internal reference

[0069] S2.2. Perform scene calibration. Use a calibration cloth to capture images based on average heights, and then mark multiple pixel coordinate reference points in the screenshots. Specifically, the average height includes the average height of children, the average height of teenagers, and the average height of adults. The number of pixel coordinate reference points should be no less than four.

[0070] In any feasible embodiment, the average height of children is 1 meter, the average height of teenagers is 1.2 meters, and the average height of adults is 1.5 meters.

[0071] S2.3. Generate a two-dimensional world coordinate system according to the actual scene; identify the world coordinate system corresponding to the corner coordinates of the calibration cloth in the coordinate system; and store the world coordinates of the calibration cloth.

[0072] That is, a two-dimensional pseudo world coordinate system is generated, the locations of the calibration cloth reference points are marked and stored.

[0073] S2.4. Solve for extrinsic parameter conversion parameters. Specifically, based on the known camera intrinsic parameters and distortion coefficients, the known calibration pixel coordinates and world coordinates, solve for the camera extrinsic parameter rotation vector and translation vector; solve for the homography matrix from the correction image to world coordinates; and transform the correction image coordinates to world coordinates using the homography.

[0074] More specifically, the camera extrinsic rotation vector and translation vector are solved using the pnp method;

[0075] In this step, the process of solving the camera extrinsic rotation vector and translation vector includes:

[0076] Select four 3D points in the world coordinate system (X a ,Y a ,Z a )(X b ,Y b ,Zb )(X c ,Y c ,Z c )(X d ,Y d ,Z d ) and the 2D points (u') at the four corners of the calibration cloth a ,v' a ,1' a )(u' b ,v' b ,1' b )(u' c ,v' c ,1' c )(u' d ,v' d ,1' d ) corresponds to, assuming that the 3D point P a =(X a ,Y a ,Z a ) The coordinates of the corresponding 2D point in the pixel coordinate system are m a =(u a ,v a ,1 a ). Known camera internal parameters According to the perspective projection formula: s is a scaling factor used to balance the loss of depth information during the planar projection from 3D space to 2D images. is the pixel coordinate of the 2D point image, u is the horizontal coordinate, and v is the vertical coordinate. A is the intrinsic parameter matrix calibrated by the camera P a is the coordinate of the 3D point, R is the rotation matrix, t=(t x ,t y ,t x ) T For the translation vector, the final camera extrinsic parameters are calculated by establishing the projection relationship equation and selecting an appropriate solution method (such as direct linear transformation DLT or iterative optimization method).

[0077] According to the new intrinsic and extrinsic parameters of the camera, the calibration cloth is converted to the coordinates of the pseudo-corrected calibration cloth. Given the corrected coordinates of the calibration cloth and the corresponding pseudo-world 2D coordinates, the homography matrix from the corrected image coordinates to the pseudo-world coordinates is calculated. The formula for the homography matrix is: The 2D pixel coordinate u is the horizontal coordinate, and v is the vertical coordinate corresponding to the X and Y coordinates of the 3D point. The inverse solution is used to obtain the homography matrix H. The matrix is ​​a 3x3 matrix that contains the set transformation information from the source point coordinates to the target coordinates. The element h 11 and h 12 Related to scaling and rotation, h13 and h 23 It mainly involves the translation part.

[0078] The correction image can be converted to pseudo-world 2D coordinates using the homography matrix, that is, the pixel coordinates of the head are converted to the 2D coordinates of the head after correction.

[0079] Coordinate transformation formula:

[0080] u i =h 11 x i +h 12 y i +h 13 , v i =h 21 x i +h 22 y i +h 23 ,ω i =h 31 x i +h 32 y i +h 33 . The element h 11 and h 12 Related to scaling and rotation, h 13 and h 23 It mainly involves the translation part, x i ,y i is the source coordinate point, u i is the horizontal coordinate after transformation, v i is the vertical coordinate after transformation.

[0081] The proportional relationship is calculated based on the actual distance between the calibration points and the pixel distance of the calibration points in the two-dimensional coordinate system. The two-dimensional coordinate pixel represents the actual number of centimeters.

[0082] The density area, speed area, queuing area, and out-of-bounds area are marked in the correction image and converted into the two-dimensional space coordinate system of the head through conversion parameters. According to the proportional relationship, the area parameters of the density area, the total length parameters of the queuing area, and the conversion parameters between the pixel distance and the actual distance of the speed area are calculated.

[0083] S3. Read the camera configuration file to be detected. The configuration file includes the stream address, camera code, and pre-allocated GPU index. The coordinate conversion model and region parameters are loaded according to the camera number. The camera stream is read, the video stream is hardware-decoded, and frames are extracted and stored in a queue. The video frames in the queue are read, pedestrian detection and tracking are performed, and pedestrian IDs are generated. The head pixel coordinates and age information are bound to the IDs; the age information of the person is identified based on the detection model. This means that a conversion model is obtained for converting the head pixel coordinates of each age group into the head's two-dimensional spatial coordinate system. The region parameters are then calibrated using the conversion model, with the region parameters being based on the two-dimensional spatial coordinate system of an adult's head.

[0084] This step is the camera decoding frame extraction inference stage.

[0085] According to the number of cameras to be detected, the corresponding process is assigned to read the corresponding configuration file.

[0086] The configuration file includes GPU index, camera number, stream address, detection model address, coordinate conversion model parameters and region parameters.

[0087] Call the GPU according to the GPU index to hard-decode the video stream and store it in the queue, then load the inference network model, load the coordinate transformation model and region parameters.

[0088] The implementation of this step may include the following existing technologies. For example, the two-dimensional coordinate information of the human head space is detected through an inference model. The multi-target tracking algorithm uses the deepsort algorithm. The detected target is re-identified and the target's features are extracted. The Hungarian algorithm is used to associate the tracking box tracks in the previous frame with the detection box detections in the current frame. The cost matrix is ​​calculated using appearance information and Mahalanobis distance, or IOU. The position of the track at the next moment is predicted by Kalman filtering, the predicted position is updated based on detection, and the ID of the detected person is output.

[0089] S4. Based on the pedestrian's age, the head pixel coordinate information is input into the coordinate conversion model at different heights and the actual spatial position is calculated. That is, the age group of the person is identified and the person is input into different head two-dimensional spatial coordinates to calculate their actual position.

[0090] In this step, the calibration system calibrates different heights. The homography matrix parameters for converting pixel coordinates of 1 meter for children, 1.2 meters for teenagers, and 1.5 meters for adults into 2D world coordinates can be determined to calculate the actual spatial position. The implementation of this step includes:

[0091] S4.1. Based on the identified pedestrian's age information, convert the pedestrian's head pixel coordinates to the corrected head pixel coordinates using the camera's intrinsic parameters. Given the camera distortion coefficient, for any point (x, y) in the image, perform distortion correction using the inverse of the above distortion formula. The corrected coordinates are:

[0092] The same applies to tangential distortion.

[0093] S4.2. The corrected head pixel coordinates are brought into the coordinate transformation model of the corresponding age and transformed into the two-dimensional head space coordinate system, i.e., the actual space coordinates, according to the homography matrix.

[0094] Conversion formula: u i =h 11 x i +h 12 y i +h 13 , v i =h 21 x i +h 22 y i +h 23 ,ω i =h 31 x i +h 32 y i +h 33

[0095] S5. The positions are uniformly converted to the parameter coordinates of the adult area; it is determined whether the corresponding parameter calculation valid area is entered, and the passenger flow parameters are calculated; that is, according to the actual spatial position, the corresponding passenger flow parameters are calculated in the area and the results are output. Among them, the area includes density area, speed area, queuing area, and out-of-bounds area. The above process is implemented using the ray intersection algorithm. Specifically, the point to be judged emits a ray in any direction, and then counts the number of times this ray crosses the polygon boundary. If the number of intersections is an odd number, the point is inside the polygon; if the number of intersections is an even number, the point is outside the polygon. More specifically, the vertex sequence of the polygon is p = {p 0, p 1, p 2, ....p n-1}, p represents the set of vertex sequences of polygons, p 0, p 1, p 2, ....p n-1 Represent the coordinates of each vertex of the polygon, where p i =(x i, y i ), the point to be judged is m=(x mi, y mi), a ray is emitted from point m in the horizontal direction (the square of the x-axis), and the ray equation is y=y m (Because it is a horizontal ray). For each edge pipi of the polygon +1 (When i=n-1, the next vertex p0 is used to form a closed polygon). The straight line equation of the edge can be expressed in two-point form as: Calculate the intersection of the ray and the edge, let y = y m , solve for the value of x and get To determine whether the intersection is valid, the x coordinate of the intersection must be greater than the x coordinate of point m, that is, x>x m At the same time, the intersection point must be on the edge pipi +1 This can be determined by judging whether the y coordinate of the intersection point is within the y coordinate range of the two end points of the edge, that is, min(y i ,y i+1 )≤y m ≤max(y i ,y i+1 Count the number of intersections, traverse all edges of the polygon, and calculate the validity of the intersection points as described above. If a valid point is found, the number of intersections is increased by 1. If the number of intersections is odd, then point m is inside the polygon, and if it is even, then it is outside.

[0096] Average density calculation: The coordinate set of the density area and the area S of the density area have been obtained. The passenger flow density parameters are calculated once per second. The density is calculated once every 25 frames. The number of people n whose coordinates are within the density area detected and tracked in the current frame is counted. The passenger flow density is calculated based on the ratio of the effective density area S. The formula is: avgd = n / s.

[0097] Heatmap calculation: Data point definition: When a pedestrian enters a density zone, the 2D coordinates of the pedestrian's head are used as the x and y coordinates of the data point. The number of times each x and y coordinate is collected per second. The sum of the x and y counts over 10 seconds is divided by 10 and averaged to obtain the current average count for each x and y coordinate, which is the density value. This completes data point collection. A thermal grid is created using a 100-cm spacing proportional to the distance between the 2D pixel points of the pedestrian's head. The data point radius is set using a 30-cm spacing proportional to the distance between the 2D pixel points of the pedestrian's head. Given the data point location, density value, influence radius, and grid parameters, grid heat is calculated. The data point is traversed to determine its position within the grid. The heat contribution of each grid cell is calculated using a Gaussian function based on the data point's location and density value, as well as the distance between the grid cell and the data point. The heat contributions of the grid cells are accumulated to obtain the final heat value, which is then normalized based on the historical maximum and minimum values. The color gradient setting divides the gradient into 6 levels: 0.1 dark blue, 0.2 light blue, 0.3 green, 0.5 yellow, 0.75 orange, and 1.0 red.

[0098] Speed ​​calculation: Real-time speed, calculate personal speed once per second. The initial tracking generates the ID formula represented by i. When the id's head space 2D coordinates enter the head space 2D coordinates of the speed area, the current id's head space 2D pixel coordinates are recorded. After every 25 frames, the pixel distance between the id's latest pixel coordinates and the previously recorded pixel coordinates is calculated. Pixel values, through the proportional relationship (p / d): p is the pixel value in the two-dimensional head space, representing the actual d centimeters, divided by (p / d) to get the actual speed centimeters, and then divided by 100 to get the current ID actual speed meters. The real-time speed calculation formula is: Average speed: sum the speeds of all pedestrians in the speed zone and divide by the number of people in the speed zone, n. The formula is: avgs = ∑ism / n. Average linear speed: calculate the pixel distance between the last coordinate of each person appearing in the speed zone and the coordinate of the first entry into the speed zone, convert it to the actual walking distance, and divide it by the difference between the last appearance time and the first appearance time to get the linear speed of each person. where t e represents the first entry time, t s represents the time of first entry. The average linear speed avgxs = ∑ismx / nx is calculated by summing the linear speeds within the time period and dividing it by the number of people, where ismx is the linear speed of the individual and nx is the number of people passing through.

[0099] Calculate the queue parameters and calibrate the generated queue area. The queue area is divided into single-channel straight-line channels and S-shaped and Z-shaped channels. S-shaped and Z-shaped channels need to be recorded as a collection of multiple straight-line channels. Each straight-line channel is a queue area. The length of the queue area can be calculated through the proportional relationship. The actual queue length is the pixel difference between the coordinates xa, ya of the first person in the queue and the coordinates xb, yb of the last person in the queue, which is converted into the actual distance. The actual queue length for the current channel is dm = dcm / 100. For an S-shaped channel, the actual queue length for multiple straight lines is summed to obtain sumdm = ∑dm. To calculate the expected queue time, t, divide the queue length by the average pedestrian speed (avgs in the speed calculation): t = sumdm / avgs.

[0100] Cross-border calculation: To determine whether a pedestrian has entered the cross-border area, it is necessary to count the pedestrian's entry time. The first time a pedestrian enters, a loudspeaker notification warning is called through the serial port protocol. The number of warnings is 3, and the alarm record is retained in the database. If the person leaves, the alarm is eliminated and the alarm record is deleted. If the entry time is as long as 1 minute and there are 6 consecutive warnings, the database alarm record warning is upgraded and the relevant staff is notified to take measures. If the person leaves, the alarm is eliminated and the alarm record is deleted.

[0101] The present invention provides a three-layer network passenger flow parameter calculation method and calibration system. Compared with the existing technology, the method divides the crowd into three stages according to their height, namely children, teenagers, and adults, and determines the average height of each stage. The average height of each stage is used as a two-dimensional spatial coordinate system (i.e., a two-dimensional spatial coordinate system of the head), and relevant calibration tools and parameter calibration schemes are customized. The pixel coordinates and the conversion parameters of the head two-dimensional spatial coordinate system and the regional parameters based on the head two-dimensional spatial coordinate system are calibrated, including parameters such as effective density area, speed area, out-of-bounds area, and queue area. In target detection, the first step is to detect whether it is a child, teenager, or adult, and then use the pixel coordinates of the person's head to convert them into the two-dimensional spatial coordinate system of the head of children, teenagers, and adults through the conversion parameters to calculate their actual position. Based on the actual position, the regional parameters are brought in to further calculate multiple passenger flow parameters including population density, speed, queue length, duration, whether out-of-bounds abnormal behavior occurs, etc. The calibration of two-dimensional spatial coordinate systems at different heights is realized. The detection and calculation of real-time passenger flow parameters of the three-layer network are realized.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-layer network passenger flow parameter calculation method, characterized in that: The following steps are involved: Use the calibration device to calibrate the camera to be tested and take a screenshot; Use the parameter calibration system combined with the screenshot to calibrate the coordinate transformation model and area parameters, and save the calibration parameters and the corresponding camera number; The parameter calibration system is used in combination with the screenshot to calibrate the coordinate transformation model and area parameters, and save the calibration parameters and the corresponding camera number, including: Capture multiple sets of calibration plate images at different positions and angles under the camera; identify the corner points of the calibration plate in the image; calculate the camera intrinsic parameter matrix and distortion coefficient; and generate the correction image and the new internal parameters after correction; The method captures multiple sets of calibration plate images at different positions and angles under the camera; identifies the corner points of the calibration plate in the image; calculates the camera intrinsic parameter matrix and distortion coefficient; and generates a correction image and a new corrected intrinsic parameter, including: Assume that the coordinates of the corner points of the calibration plate in the space coordinate system are (X a ,Y a ,Z a ), the coordinates in the camera coordinate system are (X b ,Y b ,Z b ), the transformation relationship from the space coordinate system to the camera coordinate system is represented by the rotation matrix R and the translation vector t, that is, The tangential distortion formula is: x 畸变 =x+(2*p1*x*y+p2*(r 2 +2*x 2 )),y 畸变 =y+(p1*(r 2 +2*y 2 )+2*p2*x*y), Distortion correction formula: x=x 畸变 -2p1x 畸变 y 畸变 +p2r 2 +2x 畸变 2 , Where p1 and p2 are tangential distortion coefficients; r 2 =x 2 +y 2 ; x, y are the coordinates of the pixel points in the image without considering the distortion, x 畸变 ,y 畸变 is the pixel coordinate after distortion, p1, p2 are the tangential distortion coefficients, which are obtained through camera calibration and describe the degree of tangential distortion; r represents the distance from the pixel x, y to the center of the image, and r 2 =x 2 +y 2 , The radial distortion formula is: x 畸变 =x(1+k1*r 2 +k2*r 4 +k3*r 6 ),y 畸变 =y(1+k1*r 2 +k2*r 4 +k3*r 6 ); Distortion correction: Given a known distortion coefficient, for any point (x, y) in the image, distortion correction is performed by inverse operation of the above distortion formula. The corrected coordinates are: Among them, x, y are the coordinates of the pixel points in the image without considering the distortion, x 畸变 ,y 畸变 is the pixel coordinate after distortion, k1, k2, k3 are radial distortion coefficients, k1 coefficient is mainly used to describe the first-order radial distortion. If k1>0, the lens usually shows pincushion distortion, if k1<0, the lens usually shows barrel distortion, k2, k3 are high-order distortion coefficients, used to more accurately describe complex distortion conditions, used for distortion correction; r represents the distance from the pixel point x, y to the center of the image, and r 2 =x 2 +y 2 , Assume the coordinates in the pixel coordinate system are (u, v), and introduce the focal length f χ 、f y , image center coordinates (u0, v0) and pixel offset coefficient c, the conversion relationship is where s χ and s y is the number of pixels per unit length; The least squares optimization algorithm is used to solve the internal parameter matrix The various parameters in; Perform scene calibration by taking screenshots at the average height using a calibration cloth, and then marking multiple pixel coordinate reference points in the screenshots. Generate a two-dimensional world coordinate system according to the actual scene; identify the world coordinate system corresponding to the corner coordinates of the calibration cloth in the coordinate system; store the world coordinates of the calibration cloth; Solve for external parameters and conversion parameters; Read the configuration file of the camera to be detected, read the camera code stream, perform hardware decoding on the video code stream, extract frames and store them in a queue; then read the video frames in the queue, detect and track pedestrians, generate pedestrian IDs, and bind the head pixel coordinate information and age information to the ID; Based on the pedestrian's age information, the head pixel coordinate information is brought into the coordinate transformation model at different heights to calculate the actual spatial position; The method of converting the head pixel coordinate information into coordinate conversion models at different heights according to the pedestrian's age information to calculate the actual spatial position includes: According to the age information of the identified pedestrian, the pixel coordinates of the pedestrian's head are converted to the corrected head pixel coordinates through the camera's intrinsic parameters. Given the camera distortion coefficient, for any point (x, y) in the image, the corrected coordinates are: The same applies to tangential distortion; k1, k2, and k3 are radial distortion coefficients; r represents the distance from the pixel point x, y to the center of the image; The corrected head pixel coordinates are brought into the coordinate transformation model of the corresponding age and transformed into the head two-dimensional space coordinate system, i.e., the actual space coordinates, according to the homography matrix; According to the actual spatial position, reverse calculation is performed to the area, the corresponding passenger flow parameters are calculated and the results are output.

2. A three-layer network passenger flow parameter calculation method according to claim 1, characterized in that: The method of back-calculating the area based on the actual spatial position, calculating the corresponding passenger flow parameters and outputting the results includes: the area includes one or more of a density area, a speed area, a queuing area, and an out-of-bounds area.

3. A three-layer network passenger flow parameter calculation method according to claim 1, characterized in that: The configuration file includes one or more of: a gpu index, a camera number, a code stream address, a detection model address, coordinate conversion model parameters, and area parameters.

4. A calibration system for calculating passenger flow parameters of a three-layer network, characterized in that: Used in the steps of a three-layer network passenger flow parameter calculation method as claimed in any one of claims 1 to 3, comprising: a first calibration component and a second calibration component; The first calibration assembly includes a control structure, a calibration bracket, a first motor, a second motor, a third motor, a calibration plate, and a walking structure; the bottom end of the calibration bracket is connected and fixed to the walking structure; the top end of the calibration bracket is connected and fixed to the first motor; the first motor is connected to the second motor via a first transmission rod; the second motor is connected to the third motor via a second transmission rod; and the third motor is connected and fixed to the calibration plate; The second calibration component includes a remote control, a structural support, a fourth motor, a fifth motor, a sixth motor, a seventh motor, an eighth motor, a ninth motor calibration cloth and a second walking structure.

Images