Personnel positioning method based on monocular camera and building three-dimensional model

Abstract

The invention relates to the technical field of computer vision, in particular to a personnel positioning method based on a monocular camera and a building three-dimensional model. The personnel positioning method comprises the following steps: loading a three-dimensional model of a building and establishing a reference coordinate system, calibrating position coordinates and attitude parameters of a monocular camera in the reference coordinate system, and calibrating internal parameters and external parameters of the monocular camera; performing Gaussian filtering denoising, illumination compensation and distortion correction preprocessing on the original image acquired by the monocular camera in sequence; performing personnel target real-time detection by adopting a convolutional neural network target detection model, and outputting personnel bounding box parameters and confidence; and mapping the two-dimensional person bounding box into a three-dimensional space by using a camera pose matrix, and calculating the three-dimensional position coordinates of the person by combining a building three-dimensional model as a boundary constraint condition. The personnel positioning method is low in deployment cost and high in positioning precision.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method for locating people based on a monocular camera and a 3D building model. Background Technology

[0002] With the rapid development of smart building and Internet of Things technologies, the demand for indoor personnel positioning technology in fields such as security monitoring, personnel management, and emergency rescue has increased dramatically.

[0003] Currently, methods used for personnel positioning include WiFi signal strength (RSSI) based positioning technology, Bluetooth beacon based positioning technology, ultra-wideband (UWB) based positioning technology, computer vision based positioning technology, and multi-sensor fusion based positioning technology.

[0004] However, these positioning methods often suffer from high costs and poor positioning accuracy. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this application is to provide a method for personnel positioning based on a monocular camera and a 3D building model, so as to achieve high-precision positioning at low cost.

[0006] To achieve the above objectives, this application provides a method for personnel positioning based on a monocular camera and a 3D building model, comprising: loading the 3D building model and establishing a reference coordinate system, calibrating the position coordinates and attitude parameters of the monocular camera in the reference coordinate system, and calibrating the intrinsic and extrinsic parameters of the monocular camera;

[0007] The raw images acquired by the monocular camera are sequentially preprocessed by Gaussian filtering for noise reduction, illumination compensation, and distortion correction.

[0008] A convolutional neural network target detection model is used for real-time detection of personnel targets, and the parameters and confidence scores of personnel bounding boxes are output. The two-dimensional personnel bounding boxes are mapped to three-dimensional space using the camera pose matrix, and the three-dimensional position coordinates of the personnel are calculated by combining the three-dimensional building model as boundary constraints.

[0009] Furthermore, the specific steps of loading the 3D building model and establishing a reference coordinate system, calibrating the position coordinates and attitude parameters of the monocular camera in the reference coordinate system, and calibrating the intrinsic and extrinsic parameters of the monocular camera include:

[0010] Choose the geometric center of the building or a specific landmark as the origin of the coordinate system;

[0011] The X and Y axes are determined by the direction of the building's main axis.

[0012] The vertical upward direction determines the Z-axis direction;

[0013] The intrinsic parameter matrix and distortion coefficients were obtained using the Zhang Zhengyou calibration method;

[0014] The extrinsic parameter matrix is ​​solved by using at least four non-coplanar control points.

[0015] Furthermore, the specific steps for performing Gaussian filtering for noise reduction, illumination compensation, and distortion correction preprocessing on the raw images acquired by the monocular camera include:

[0016] Gaussian filtering is performed using Gaussian filter kernels with dimensions of 3×3 or 5×5.

[0017] Illumination compensation is performed using histogram equalization or gamma correction.

[0018] Radial and tangential distortions are eliminated based on calibration parameters.

[0019] Furthermore, the specific steps for using a convolutional neural network target detection model to perform real-time personnel target detection and output personnel bounding box parameters and confidence scores include:

[0020] The YOLOv7 model was used as the convolutional neural network object detection model, with an input image size of 640×640 pixels.

[0021] The confidence threshold for personnel detection was set to 0.5-0.7;

[0022] The non-maximum suppression threshold is set to 0.4-0.6;

[0023] The output bounding box format is [x_center,y_center,width,height], where x_center is the center coordinate of the bounding box on the X-axis, y_center is the center coordinate of the bounding box on the Y-axis, width is the width of the bounding box, and height is the height of the bounding box.

[0024] Furthermore, the specific steps for mapping the two-dimensional personnel bounding box to three-dimensional space using the camera pose matrix, and combining it with the three-dimensional building model as boundary constraints, to calculate the three-dimensional position coordinates of the personnel include:

[0025] The two-dimensional image coordinates of the foot contact point are determined based on the bottom center point of the personnel boundary frame;

[0026] Establish the mapping relationship from the image coordinate system to the world coordinate system using the camera intrinsic and extrinsic parameter matrices;

[0027] Based on the depth constraint, the three-dimensional position coordinates of the personnel are calculated through the intersection of the ray and the ground plane.

[0028] Furthermore, the specific steps for mapping the two-dimensional personnel bounding box to three-dimensional space using the camera pose matrix, and combining it with the three-dimensional building model as boundary constraints, to calculate the three-dimensional position coordinates of the personnel include:

[0029] Check whether the calculated three-dimensional position coordinates of the personnel are located inside the building space;

[0030] Exclude invalid locations within the solid structure of walls and columns;

[0031] Detection results for locations that are abnormal are corrected or removed.

[0032] Furthermore, the method also includes:

[0033] A filter is used to smooth and optimize the three-dimensional position coordinates of personnel, removing position noise.

[0034] Furthermore, the specific steps for smoothing and optimizing the three-dimensional position coordinates of the personnel using a filter to remove position noise include:

[0035] Establish a motion state model of personnel and a Kalman filter, and predict the next state vector and the next position coordinates based on the personnel's state variables and process noise covariance matrix in each frame. The state variables include position coordinates and velocity.

[0036] Based on the actual position observation vector, predicted next state vector, predicted next position coordinates, and observation noise covariance matrix of the monocular camera, the Kalman gain is updated to obtain the filtered state vector and position coordinates. The update frequency is consistent with the camera frame rate.

[0037] Furthermore, the state transition matrix of the Kalman filter is:

[0038] F = [[1,0,0,dt,0,0],[0,1,0,0,dt,0],[0,0,1,0,0,dt],[0,0,0,1,0,0],[0,0,0,0,1,0],[0,0,0,0,1]], where dt is the time interval.

[0039] Furthermore, the method also includes:

[0040] Assign a unique ID to each detection target;

[0041] The Hungarian algorithm is used to associate targets in adjacent frames, and the association cost is Euclidean distance.

[0042] The timeout threshold for trajectory loss is set to 3 seconds.

[0043] Furthermore, the three-dimensional building model satisfies:

[0044] OBJ, PLY, FBX, or IFC format

[0045] The geometric error is less than 0.1 meters;

[0046] Includes semantic tags for entity structures such as walls, floors, and ceilings.

[0047] Furthermore, the monocular camera has a resolution of no less than 1920×1080 pixels, a frame rate of no less than 25fps, a field of view of 60°-90°, and supports autofocus and auto exposure functions.

[0048] The personnel positioning method based on a monocular camera and a 3D building model provided in this application only requires a regular monocular camera and computing equipment, without the need to deploy dedicated hardware such as UWB anchor points and Bluetooth beacons, which greatly reduces deployment costs compared to traditional solutions.

[0049] The personnel positioning method based on a monocular camera and a 3D building model provided in this application reduces positioning errors and improves positioning accuracy by using geometric constraints of the building model and ground height constraints.

[0050] The personnel localization method based on a monocular camera and a 3D building model provided in this application uses the YOLOv7 model, which has a high detection speed. Combined with Kalman filtering optimization, the overall system latency is low, which can meet the needs of emergency response.

[0051] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing this application. Attached Figure Description

[0052] The accompanying drawings are provided to further illustrate the present application and form part of the specification. Together with the embodiments of the present application, they serve to explain the present application but do not constitute a limitation thereof. In the drawings:

[0053] Figure 1 This is a flowchart illustrating the personnel positioning method based on a monocular camera and a 3D building model according to Embodiment 1 of this application.

[0054] Figure 2 This is a flowchart of real-time human target detection using the YOLOv7 model in Embodiment 1 of this application;

[0055] Figure 3 This is a flowchart illustrating the calculation of the three-dimensional position coordinates of a person in Embodiment 1 of this application;

[0056] Figure 4 This is a flowchart illustrating the smoothing optimization process using filters.

[0057] Figure 5This is a flowchart illustrating the personnel positioning method based on a monocular camera and a 3D building model according to Embodiment 2 of this application. Detailed Implementation

[0058] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0059] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0060] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0061] It should be noted that the terms "one" and "multiple" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "Multiple" should be understood as two or more.

[0062] Terminology Explanation:

[0063] A monocular camera, a digital camera device containing only one lens, lacks direct stereo depth information compared to binocular or multi-lens cameras.

[0064] A 3D architectural model is a digital 3D representation of a building's geometry and spatial relationships, such as a BIM model or a point cloud model.

[0065] YOLOv7, You Only Look Once version 7, is a real-time object detection deep learning algorithm.

[0066] Camera calibration is the process of determining the camera's intrinsic parameters (focal length, principal point) and extrinsic parameters (position, attitude).

[0067] The Kalman filter, a recursive digital filter, is used to estimate the state of a dynamic system from a series of noisy observations. Zhang Zhengyou's calibration method, a camera calibration method based on a planar checkerboard grid, can simultaneously solve for intrinsic parameters and distortion parameters.

[0068] Non-Maximum Suppression (NMS) is a post-processing algorithm used to remove duplicate bounding boxes in object detection.

[0069] Example 1

[0070] One embodiment of this application provides a method for personnel localization based on a monocular camera and a 3D building model, used to locate personnel inside an office building. Figure 1 This is a flowchart illustrating the personnel localization method based on a monocular camera and a 3D building model according to Embodiment 1 of this application. The following will refer to... Figure 1 This application provides a detailed description of the personnel localization method based on a monocular camera and a 3D building model, including:

[0071] Step S101: Load the 3D building model and establish a reference coordinate system. In the reference coordinate system, calibrate the position coordinates and attitude parameters of the monocular camera, and calibrate the intrinsic and extrinsic parameters of the monocular camera.

[0072] In this embodiment, the hardware configuration used is as follows:

[0073] Monocular camera: Hikvision DS-2CD2625FWD-IZS, resolution 1920×1080, frame rate 25fps, field of view 60°

[0074] Computing equipment: An industrial computer equipped with an NVIDIA GTX 1660Ti GPU, 16GB of RAM, and a 256GB SSD for storage.

[0075] 3D building model: The office building BIM model was created using Autodesk Revit and exported as OBJ format.

[0076] In this embodiment, the 3D model file building_model.obj of the office building is loaded first. The model contains complete geometric information such as floor structure, walls, and columns, and the model accuracy is 0.05 meters.

[0077] Then, the geometric center of the office building is chosen as the origin O(0,0,0) of the world coordinate system (base coordinate system), with the east-west direction of the office building as the positive X-axis.

[0078] The north-south direction is the positive Y-axis, and the vertical upward direction is the positive Z-axis.

[0079] The camera is installed in a corner of the conference room, 2.8 meters above the ground. The world coordinates of the installation location are (12.5, 8.3, 2.8) and the attitude parameters are (pitch angle α, yaw angle β, roll angle γ).

[0080] Camera intrinsic parameters were calibrated using a 50mm×50mm black and white checkerboard pattern. Fifteen calibration images were taken from different angles, and the camera intrinsic parameter matrix was calculated using Zhang Zhengyou's calibration algorithm.

[0081] K=[[1068.2,0,960.0],[0,1071.5,540.0],[0,0,1]];

[0082] Radial distortion coefficients: k1 = -0.1854, k2 = 0.0286;

[0083] Tangential distortion coefficients: p1 = 0.0008, p2 = -0.0012;

[0084] Four spatial control points were selected within the office space for external parameter R calibration:

[0085] Control point 1: (10.0, 6.0, 0.0) -> Image coordinates (356, 789)

[0086] Control point 2: (15.0, 6.0, 0.0) -> Image coordinates (1205, 805)

[0087] Control point 3: (10.0, 10.0, 0.0) -> Image coordinates (298, 485)

[0088] Control point 4: (15.0, 10.0, 0.0) -> Image coordinates (1089, 502)

[0089] The extrinsic parameter matrix R and translation vector t are obtained by solving the PnP algorithm, and a complete transformation relationship from the image coordinate system to the world coordinate system is established.

[0090] Step S102: Gaussian filtering for noise reduction, illumination compensation, and distortion correction preprocessing are sequentially performed on the original images acquired by the monocular camera. In this embodiment, the monocular camera continuously acquires images at a frame rate of 25fps, and the original image resolution is 1920×1080. First, distortion correction is performed, and the distortion coefficients obtained in step S101 are used to eliminate image deformation.

[0091] Then, a 3×3 Gaussian kernel is used for image denoising: G(x,y)=(1 / 16)*[[1,2,1],[2,4,2],[1,2,1]]; then, for uneven illumination, contrast-limited adaptive histogram equalization (CLAHE) is used for illumination correction, with the tile size set to 8×8 and the contrast limit threshold set to 2.0.

[0092] In some other implementations, gamma correction can also be used for optical noise compensation.

[0093] Finally, the image size was adjusted to 640×640 and used as input to the convolutional neural network object detection model.

[0094] Step S103: Use a convolutional neural network target detection model to perform real-time detection of personnel targets, and output the personnel bounding box parameters and confidence scores;

[0095] Figure 2 This is a flowchart of real-time person target detection using the YOLOv7 model in Embodiment 1 of this application, referred to... Figure 2 When a 640×640 image is input into the YOLOv7 model backbone network, it will pass through the feature extraction layer, then perform FPN (Feature Pyramid) feature fusion, then use the detection head to regress and calculate the bounding box and confidence score, then process it through NMS (Non-Maximum Suppression) to output the person bounding box, and then output the detection result through coordinate format conversion and confidence score filtering.

[0096] In the embodiments of this application, the convolutional neural network object detection model used is the YOLOv7 model. A pre-trained YOLOv7 model file, yolov7.pt, is loaded. The model is trained on the COCO dataset, which includes a people category (class_id = 0). A pre-processed 640×640 image is input into the network, and the detection result is obtained through inference. The detection confidence threshold is set to 0.6, and the NMS threshold is set to 0.5.

[0097] Typical detection output example:

[0098] Detection box 1: [0.3125, 0.4687, 0.1875, 0.5625], confidence level 0.89, category "personnel";

[0099] Detection box 2: [0.7031, 0.5156, 0.1406, 0.4219], confidence level 0.76, category "personnel";

[0100] It should be noted that the confidence threshold for human detection can be set to 0.5-0.7; the non-maximum suppression threshold can be set to 0.4-0.6.

[0101] The bounding box format is [x_center, y_center, width, height], where x_center is the center coordinate of the bounding box on the X-axis, y_center is the center coordinate of the bounding box on the Y-axis, width is the width of the bounding box, and height is the height of the bounding box.

[0102] Convert normalized coordinates to pixel coordinates:

[0103] Personnel 1: Center point (200, 300), width 120, height 360, foot position (200, 480).

[0104] Personnel 2: Center point (450, 330), width 90, height 270, foot position (450, 465).

[0105] Step S104: Use the camera pose matrix to map the 2D personnel bounding box into the 3D space, and combine the 3D building model as the boundary constraint to calculate the 3D position coordinates of the personnel.

[0106] Figure 3 This is a flowchart illustrating the calculation of the three-dimensional position coordinates of a person in Embodiment 1 of this application. (Refer to...) Figure 3 The system determines the pixel points of the person's feet based on the camera's optical center D and the bounding box. Then, it establishes a three-dimensional ray equation and solves for the person's position by determining the intersection point of the ray with the ground surface. Finally, it performs constraint detection on the person's position based on the three-dimensional building model.

[0107] In this embodiment, for each detected person, the image projection of the foot landing position is first taken as the center point of the bottom of the bounding box. Taking person 1 as an example, the foot pixel coordinates are (200, 480).

[0108] Establish the three-dimensional ray equation from the camera's optical center to the foot point:

[0109] The position of the camera optical center in the world coordinate system: = (12.5, 8.3, 2.8) The direction vector of the foot point in the camera coordinate system: d = K(-1) * [200, 480, 1] Transform to the world coordinate system: d_world = R * d;

[0110] Ray equation: P = D + t * d_world, where t is a parameter.

[0111] Assuming the person's feet are in contact with the ground, i.e., the Z-coordinate is 0, solve:

[0112] 0 = 2.8 + t * d_world[2];

[0113] t = -2.8 / d_world[2];

[0114] Substituting t into the ray equation, we obtain the position of the person in the world coordinate system: P1 = (11.2, 9.1, 0.0).

[0115] Perform a building model constraint check on the calculation results: determine whether the location (11.2, 9.1, 0.0) is located within the building's interior space. By querying the 3D model, it is confirmed that the location is within the accessible area of ​​the conference room, and the location is valid.

[0116] When the calculated personnel location is an invalid location inside a solid structure such as a wall or column, it needs to be corrected or excluded.

[0117] Step S105: Use a filter to smooth and optimize the three-dimensional position coordinates of the personnel to remove position noise;

[0118] Figure 4 This is a flowchart illustrating the smoothing optimization process using filters, refer to... Figure 4 Based on the motion model F and process noise, state prediction is performed. For each frame, based on position observations, a Kalman filter prediction-update loop is executed using the state vector. Specific steps include:

[0119] Establish a personnel motion model and a Kalman filter. The motion model predicts the next state vector and position based on the personnel's state vector X, the state transition matrix F of the Kalman filter, and the process noise covariance matrix Q. The formula is as follows: X_pred=F*X_prev;

[0120] P_pred = F * P_prev * F^T + Q;

[0121] Where X_pred is the predicted state vector, P_pred is the predicted position, X_prev is the previous state vector, and P_prev is the previous position.

[0122] The state vector X = [x,y,z,vx,vy,vz]^T contains position (x,y,z) and velocity information (vx,vy,vz).

[0123] The state transition matrix F of the Kalman filter (dt = 0.04s, corresponding to a camera frame rate of 25fps) is as follows:

[0124] F=[[1,0,0,0.04,0,0],[0,1,0,0,0.04,0],[0,0,1,0,0,0.04],[0,0,0,1,0,0],[0,0,0,0,1,0],[0,0,0,0,0,1]]

[0125] Wherein, the process noise covariance matrix Q:

[0126] Q=diag([0.1,0.1,0.05,0.5,0.5,0.2])

[0127] in,

[0128] In this embodiment,

[0129] Initial state vector: X0 = [11.2, 9.1, 0.0, 0.0, 0.0, 0.0]^T

[0130] Initial covariance: P0 = diag([1.0, 1.0, 0.5, 2.0, 2.0, 1.0])

[0131] Observation noise covariance matrix: R = diag([0.25, 0.25, 0.1])

[0132] After prediction, the observation vector Z and observation matrix are updated based on the actual position observed by the camera, and the update frequency is consistent with the camera frame rate.

[0133] Update steps:

[0134] K=P_pred*HT*(H*P_pred*HT+R)^(-1);

[0135] X_new=X_pred+K*(ZH*X_pred);

[0136] P_new = (IK*H)*P_pred;

[0137] Where H is the observation matrix, H = [[1,0,0,0,0,0],[0,1,0,0,0,0],[0,0,1,0,0,0]], I is the identity matrix, and K is the Kalman gain.

[0138] The position results after filtering optimization are more stable, effectively suppressing position jitter caused by detection errors and geometric calculation errors.

[0139] In some other implementations, the update frequency can be set between 25-30 Hz, depending on the camera's frame rate.

[0140] Example 2

[0141] One embodiment of this application provides a method for personnel localization based on a monocular camera and a 3D building model. Figure 5 This is a flowchart illustrating the personnel localization method based on a monocular camera and a 3D building model according to Embodiment 2 of this application. The following will refer to... Figure 5 This application provides a detailed description of the personnel localization method based on a monocular camera and a 3D building model. The difference between Embodiment 2 and Embodiment 1 is as follows:

[0142] In this embodiment, the hardware configuration has been optimized. In Example 2, the hardware and building model specifications are as follows:

[0143] High-resolution camera, Dahua DH-IPC-HFW5831E-ZE, 3840×2160 (4K) resolution, 30fps frame rate, 70° field of view. High-performance computing device, equipped with NVIDIA RTX 3080 GPU, 32GB RAM, and 1TB NVMe SSD storage.

[0144] High-precision architectural model. A 3D model reconstructed using laser scanning point cloud data, with an accuracy of 0.02 meters.

[0145] In this implementation, a small target detection branch is added based on YOLOv7, and a denser anchor box setting is used to improve the detection accuracy of people at a distance. The input image resolution is increased to 1280×1280, and the detection accuracy is improved by 12% compared to a 640×640 input.

[0146] In this embodiment, the detection confidence threshold is dynamically adjusted based on image quality and ambient lighting conditions.

[0147] When there is sufficient light, the confidence threshold is set to 0.7.

[0148] When the lighting is normal: the confidence threshold is set to 0.6.

[0149] When the lighting is poor: the confidence threshold is set to 0.5.

[0150] In this embodiment, when calculating the position of a person, in addition to the center point of the feet, key points such as the person's head and shoulders are also used for position estimation, and then the final position is obtained through weighted fusion.

[0151] For example, the weight of the foot position is Weight_foot = 0.6; the weight of the head position is Weight_head = 0.3; and the weight of the shoulder position is Weight_shoulder = 0.1.

[0152] In this embodiment, the extended Kalman filter (EKF) is used to process the nonlinear motion model, taking into account complex motion states such as turning and acceleration of the person: the state vector is extended to: X=[x,y,z,vx,vy,vz,ax,ay]^T, with the acceleration components ax and ay added.

[0153] In this embodiment, positional constraints are established between adjacent frames to avoid abrupt position changes:

[0154] If |P(t)-P(t-1)|>2.0 meters, then mark it as an anomaly and enable position correction.

[0155] The performance test results of the indoor personnel positioning method in Embodiment 2 of this application are as follows:

[0156] Average positioning error: 0.28 meters (a 38% improvement compared to 0.45 meters in Example 1)

[0157] 95% confidence level error: 0.82 meters (a 32% improvement compared to 1.2 meters in Example 1)

[0158] Processing latency: 68 milliseconds (a 28% improvement compared to 95 milliseconds in Example 1)

[0159] Personnel detection accuracy: 96.3% (an improvement of 5.1% compared to 91.2% in Example 1).

[0160] Example 3

[0161] One embodiment of this application provides a method for personnel localization based on a monocular camera and a 3D building model. The difference between embodiment 3 and embodiment 1 is as follows:

[0162] In this embodiment, the personnel positioning method is used for personnel monitoring in underground parking lots.

[0163] In this embodiment, an infrared supplementary lighting device is used to improve image quality, the camera installation height is adjusted to 2.2 meters, and the field of view is increased to 90°.

[0164] In this embodiment, the ground height is set to Z = -3.0m (three underground floors).

[0165] In this embodiment, vehicle obstruction detection and continuous personnel tracking processing are also added.

[0166] Example 4

[0167] One embodiment of this application provides a method for personnel localization based on a monocular camera and a 3D building model. The difference between embodiment 4 and embodiment 1 is as follows:

[0168] In this embodiment, the personnel location method is used for personnel flow analysis in multi-level shopping malls.

[0169] In this embodiment, multiple cameras are deployed to form network coverage, and coordinate system calibration is performed between the cameras.

[0170] In this embodiment, personnel identification and cross-camera tracking algorithms are added.

[0171] In this embodiment, the NMS algorithm for target detection is optimized for densely populated scenes.

[0172] In this implementation, a hierarchical statistical model is established to calculate the distribution of people on each floor.

[0173] Example 5

[0174] One embodiment of this application provides a method for personnel localization based on a monocular camera and a 3D building model. The difference between Embodiment 5 and Embodiment 1 is as follows:

[0175] In this embodiment, the personnel positioning method is used for safety monitoring of the factory production line.

[0176] In this embodiment, a detailed three-dimensional model of the work area is established, including equipment location information.

[0177] In this embodiment, an intrusion detection function for dangerous areas is added.

[0178] In this implementation, trajectory-based abnormal behavior analysis is employed.

[0179] In this embodiment, the factory's MES system is integrated, and employee IDs and location information are associated.

[0180] In this embodiment, a multi-camera system configuration is also employed:

[0181] For large-area coverage requirements, a multi-camera network deployment is adopted:

[0182] Camera 1: Location (0,0,3.0), Coverage area A;

[0183] Camera 2: Position (20,0,3.0), Coverage area B;

[0184] Camera 3: Position (10, 15, 3.0), Coverage area C;

[0185] The overlapping areas are fused using a location information fusion algorithm.

[0186] Example 6

[0187] One embodiment of this application provides a method for personnel localization based on a monocular camera and a 3D building model. The difference between Embodiment 6 and Embodiment 1 is as follows:

[0188] In this embodiment, a 180° fisheye camera is used to expand the monitoring range. The original image is mapped to a perspective projection image using a fisheye image correction algorithm, and then a standard detection and positioning process is executed. Camera calibration requires a specialized fisheye model that includes more distortion parameters.

[0189] Example 7

[0190] One embodiment of this application provides a method for personnel localization based on a monocular camera and a 3D building model. The difference between embodiment 7 and embodiment 1 is as follows:

[0191] In this embodiment, a standard USB camera (1280×720 resolution, costing less than 50 yuan) is used, and software algorithms compensate for insufficient hardware performance. Preprocessing steps such as image enhancement and deblurring are added, and the YOLOv7 network structure is optimized to adapt to low-resolution input.

[0192] Example 8

[0193] One embodiment of this application provides a method for personnel localization based on a monocular camera and a 3D building model. The difference between embodiment 8 and embodiment 1 is as follows:

[0194] In this embodiment, the personnel localization method is deployed in a lightweight version on an ARM platform (such as NVIDIA Jetson Nano), using the YOLOv7-tiny model, with an inference speed of 15fps and power consumption of less than 10W, making it suitable for environments where GPU servers cannot be deployed.

[0195] Example 9

[0196] One embodiment of this application provides a method for personnel localization based on a monocular camera and a 3D building model. The difference between embodiment 9 and embodiment 1 is as follows:

[0197] In this embodiment, the positioning method is used for a 500-square-meter office area in an office building.

[0198] In this embodiment, performance tests were conducted on the personnel positioning method, UWB positioning method (accuracy 0.1-0.3 meters), and WiFi fingerprint positioning method (accuracy 3-5 meters) of the present application. The test time was from 9:00-12:00 am and 2:00-6:00 pm on weekdays. The number of test participants was 20-30 people present at the same time.

[0199] The test results for the performance indicators are shown in Table 1:

[0200] Table 1

[0201]

[0202] As shown in Table 1, the positioning method in this embodiment has a much lower deployment cost than the UWB positioning method, but its performance indicators, such as average positioning error, 95% confidence error, and real-time processing latency, are not significantly different from those of the UWB positioning method.

[0203] The performance test results of the embodiments of this application under different illuminations are shown in Table 2:

[0204] Table 2

[0205]

[0206] As shown in Table 2, the positioning method of this application embodiment still achieves a detection accuracy of up to 78.5% in a nighttime environment with a light intensity of <50 lux. The performance test results of this application embodiment in a multi-person scenario are shown in Table 3.

[0207] Table 3

[0208]

[0209] As shown in Table 3, the positioning method of this application embodiment has a tracking success rate of up to 83.6% in an environment of 16-20, with a latency of only 125 milliseconds.

[0210] It should be noted that when there are cases where detection is missed due to factors such as people being obstructed, insufficient lighting, or abnormal posture, multi-angle cameras can be added, the sensitivity of the detection threshold can be improved, and image preprocessing can be enhanced.

[0211] It should be noted that when positioning accuracy decreases due to camera calibration errors, building model errors, or uneven ground conditions, positioning accuracy can be improved by periodically recalibrating, improving model accuracy, and adding ground height detection.

[0212] It should be noted that when the tracking effect is poor due to people meeting, similar appearance, or fast movement, appearance features, trajectory prediction, and multi-frame association can be added to assist tracking.

[0213] The above description is merely a partial embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0214] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0215] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for personnel localization based on a monocular camera and a 3D building model, characterized in that, include: Load the 3D building model and establish a reference coordinate system. In the reference coordinate system, calibrate the position coordinates and attitude parameters of the monocular camera, and calibrate the intrinsic and extrinsic parameters of the monocular camera. The raw images acquired by the monocular camera are sequentially preprocessed by Gaussian filtering for noise reduction, illumination compensation, and distortion correction. A convolutional neural network target detection model is used for real-time detection of people targets, and the bounding box parameters and confidence scores of the people are output. The 2D bounding box of the person is mapped to 3D space using the camera pose matrix, and the 3D building model is used as the boundary constraint to calculate the 3D position coordinates of the person.

2. The personnel positioning method based on a monocular camera and a 3D building model according to claim 1, characterized in that, The specific steps of loading the 3D building model and establishing a reference coordinate system, calibrating the position coordinates and attitude parameters of the monocular camera in the reference coordinate system, and calibrating the intrinsic and extrinsic parameters of the monocular camera include: Choose the geometric center of the building or a specific landmark as the origin of the coordinate system; The X and Y axes are determined by the direction of the building's main axis. The vertical upward direction determines the Z-axis direction; The intrinsic parameter matrix and distortion coefficients were obtained using the Zhang Zhengyou calibration method; The extrinsic parameter matrix is ​​solved by using at least four non-coplanar control points.

3. The personnel positioning method based on a monocular camera and a 3D building model according to claim 1, characterized in that, The specific steps for performing Gaussian filtering for noise reduction, illumination compensation, and distortion correction preprocessing on the raw images acquired by the monocular camera include: Gaussian filtering is performed using Gaussian filter kernels with dimensions of 3×3 or 5×5. Illumination compensation is performed using histogram equalization or gamma correction. Radial and tangential distortions are eliminated based on calibration parameters.

4. The personnel positioning method based on a monocular camera and a 3D building model according to claim 1, characterized in that, The specific steps for using a convolutional neural network target detection model to perform real-time personnel target detection and output personnel bounding box parameters and confidence scores include: The YOLOv7 model was used as the convolutional neural network object detection model, with an input image size of 640×640 pixels. The confidence threshold for personnel detection was set to 0.5-0.7; The non-maximum suppression threshold is set to 0.4-0.6; The output bounding box format is [x_center,y_center,width,height], where x_center is the center coordinate of the bounding box on the X-axis, y_center is the center coordinate of the bounding box on the Y-axis, width is the width of the bounding box, and height is the height of the bounding box.

5. The personnel positioning method based on a monocular camera and a 3D building model according to claim 1, characterized in that, The specific steps for mapping the 2D personnel bounding box to 3D space using the camera pose matrix, and combining it with the 3D building model as boundary constraints, to calculate the 3D position coordinates of the personnel include: The two-dimensional image coordinates of the foot contact point are determined based on the bottom center point of the personnel boundary frame; Establish the mapping relationship from the image coordinate system to the world coordinate system using the camera intrinsic and extrinsic parameter matrices; Based on the depth constraint, the three-dimensional position coordinates of the personnel are calculated through the intersection of the ray and the ground plane.

6. The personnel positioning method based on a monocular camera and a 3D building model according to claim 5, characterized in that, The specific steps for mapping the 2D personnel bounding box to 3D space using the camera pose matrix, and combining it with the 3D building model as boundary constraints, to calculate the 3D position coordinates of the personnel include: Check whether the calculated three-dimensional position coordinates of the personnel are located inside the building space; Exclude invalid locations within the solid structure of walls and columns; Detection results for locations that are abnormal are corrected or removed.

7. The personnel positioning method based on a monocular camera and a 3D building model according to claim 1, characterized in that, The method further includes: A filter is used to smooth and optimize the three-dimensional position coordinates of personnel, removing position noise.

8. The personnel positioning method based on a monocular camera and a 3D building model according to claim 7, characterized in that, The specific steps for smoothing and optimizing the three-dimensional position coordinates of personnel using a filter to remove position noise include: Establish a motion state model of personnel and a Kalman filter, and predict the next state vector and the next position coordinates based on the personnel's state variables and process noise covariance matrix in each frame. The state variables include position coordinates and velocity. Based on the actual position observation vector, predicted next state vector, predicted next position coordinates, and observation noise covariance matrix of the monocular camera, the Kalman gain is updated to obtain the filtered state vector and position coordinates. The update frequency is consistent with the camera frame rate.

9. The personnel positioning method based on a monocular camera and a 3D building model according to claim 8, characterized in that, The state transition matrix of the Kalman filter is: F = [[1,0,0,dt,0,0],[0,1,0,0,dt,0],[0,0,1,0,0,dt],[0,0,0,1,0,0],[0,0,0,0,1,0],[0,0,0,0,1]], where dt is the time interval.

10. The personnel positioning method based on a monocular camera and a 3D building model according to claim 1, characterized in that, The method further includes: Assign a unique ID to each detection target; The Hungarian algorithm is used to associate targets in adjacent frames, and the association cost is Euclidean distance. The timeout threshold for trajectory loss is set to 3 seconds.

11. The personnel positioning method based on a monocular camera and a 3D building model according to claim 1, characterized in that, The three-dimensional building model satisfies: OBJ format, PLY format, FBX format, or IFC format Geometric error less than 0.1 meters; Includes semantic tags for entity structures such as walls, floors, and ceilings.

12. The personnel positioning method based on a monocular camera and a 3D building model according to claim 1, characterized in that, The monocular camera has a resolution of no less than 1920×1080 pixels, a frame rate of no less than 25fps, a field of view of 60°-90°, and supports autofocus and auto exposure functions.

Images