Gimbal camera deployment and automatic calibration method for high-rise building outer wall fire positioning

Abstract

The present application relates to the technical field of wisdom fire and computer vision, and particularly relates to a PTZ camera deployment and automatic calibration method for high-rise building outer wall fire positioning. The PTZ camera deployment and automatic calibration method utilizes a BIM model to accurately guide the global optimization of camera deployment positions, utilizes a robust feature matching algorithm and PnP solving to automatically establish the correlation between real-time image feature points and BIM world coordinates, and constructs the mapping relationship between PTZ camera posture parameters and external parameters. This makes it possible to obtain the accurate external parameters of the current camera without re-shooting calibration images in any real-time monitoring state, so that the fire point pixel coordinates detected in the real-time image can be automatically and high-precision converted to the BIM world coordinate system, and the three-dimensional accurate positioning of the fire point on the building facade is realized, thereby providing key information for rapid emergency response.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent fire fighting and computer vision, and particularly relates to a gimbal camera deployment and automatic calibration method for high-rise building outer wall fire positioning. BACKGROUND

[0002] With the acceleration of urbanization, high-rise / super high-rise buildings are increasing, and the fire monitoring system is one of the important systems of high-rise buildings. The current high-rise fire monitoring and positioning system is mainly aimed at electrical fires and internal fires, and uses temperature, smoke and other sensors to monitor indoors. Among them, cameras are widely used in building management and structural health monitoring due to their low cost and high efficiency. In recent years, cameras have become an economical and efficient sensor for construction management and structural health monitoring in the construction industry. The use of cameras is very effective in project progress tracking, quality assurance and safety monitoring, and can ensure real-time problem identification. In structural health monitoring, cameras can continuously monitor the building structure for a long time, which helps to discover problems in time and conduct subsequent analysis. Furthermore, the accuracy and effectiveness of structural health monitoring and construction management depend on the quality of captured images, and the deployment strategy of cameras will affect the quality of the images captured, thereby affecting the accuracy and effectiveness of the judgments made based on the quality of the captured images. In the high-rise building system, the volume is generally large, the structure is complex, and there are many obstacles (such as trees, adjacent buildings, etc.) around the high-rise building, which are easy to cause visual obstruction to the cameras deployed on the outer wall of the high-rise building. Thus affecting the quality of the images captured. In addition, the existing cameras rely on engineering experience or simple rules when deploying, and this deployment method is difficult to ensure that the deployed cameras can have complete visual coverage of the entire outer wall of the building (especially in high-rise areas and easily obstructed areas), which will lead to the situation that it is difficult to cover blind areas or redundant deployment due to covering blind areas after deployment by the existing method. Furthermore, the resolution of the existing cameras is low, making it difficult to meet the demand for high-resolution coverage of the entire outer wall of the high-rise building.

[0003] Further, in the high-rise building fire monitoring scene, the accuracy of camera calibration directly affects the accuracy of fire point positioning. At present, the camera calibration technology mainly adopts the transmission calibration method based on the calibration board. This kind of method depends on the calibration board with known set shape (such as chessboard, circular array), and then through shooting the images of the calibration board at different poses, the corresponding relationship between the pixel coordinates of the feature points (such as corner points, circle centers) and the known world coordinates is used to solve the camera parameters. Among them, the typical algorithm includes Zhang Zhengyou calibration method. However, this method has the following limitations: (1) It depends on manual arrangement of the calibration board. In the high-rise building scene, the calibration board needs to cover different heights and angles of view, which leads to the installation difficulty and high cost of the calibration board if the existing calibration method is used to calibrate the camera of the high-rise building. (2) The existing camera calibration method cannot adapt to the dynamic gimbal camera. If the gimbal camera rotates or zooms, the original external parameters will be invalid. If recalibration is performed, it will depend on manual repositioning of the calibration board or need to use a complex algorithm, which makes it difficult to operate, low in efficiency and unable to complete real-time calibration in the high-rise building scene if the existing camera calibration method is used to calibrate the gimbal camera, so that the positioning accuracy of the pictures taken by the camera in real time cannot be guaranteed. SUMMARY

[0004] In order to solve the technical problems of unreasonable deployment of the gimbal camera and inability to automatically calibrate the dynamically changing gimbal camera with high precision by using the existing calibration method in the prior art, the present application provides a gimbal camera deployment and automatic calibration method for high-rise building outer wall fire positioning.

[0005] The application adopts the following technical scheme: a gimbal camera deployment and automatic calibration method for high-rise building outer wall fire positioning, which comprises the following steps: extracting building outer wall information, establishing a world coordinate system, and triangular meshing the building outer wall. Three-dimensional modeling is performed on the obstacles around the building, and the pre-deployed position of the gimbal camera is obtained. The number of gimbal cameras that need to be deployed is set, and the maximum coverage rate and the corresponding position coordinates under the number are obtained through a heuristic optimization algorithm. The number of gimbal cameras corresponding to the maximum coverage rate is taken as the optimal deployment number, and the position coordinates of each gimbal camera corresponding to the number are taken as the optimal deployment position of each gimbal camera. The building outer wall range image taken by the gimbal camera at the optimal deployment position is taken as the reference image, and the real-time image of each gimbal camera under different postures is obtained. Feature points are labeled on the reference image, and the 2D coordinates of the feature points on the reference image are obtained. The 2D coordinates of each feature point are mapped to the world coordinate system to obtain the 3D coordinates of the feature points in the world coordinate system, and then the coordinate point pairs of the 2D coordinates and the 3D coordinates of each feature point are constructed. The reference image and the real-time image are matched, and the coordinate point pairs corresponding to the set of feature points that successfully match in the real-time image are obtained. The mapping relationship between the coordinate point pairs and the external parameters of the gimbal camera is constructed through the PnP algorithm, a plurality of external parameter data are obtained based on the real-time images of the gimbal camera under different postures, and then the mapping relationship between the posture parameters of the gimbal camera and the external parameters is constructed. The pixel coordinates of the fire points detected in the real-time image under the determined posture are converted into the world coordinates through the above mapping relationship, and then the positioning of the outer wall fire points is realized.

[0006] As a further improvement of the application, the external parameters of the gimbal camera include a rotation matrix R and a translation vector t . The 2D coordinates of any image point in the real-time image taken by the gimbal camera are defined as p i =( u i , v i ), and the corresponding 3D coordinates of the image point in the world coordinate system are P w =( X w , Y w , Z w . The mapping relationship between the coordinate point pairs of the feature points and the external parameters of the gimbal camera is as follows:

[0007] ;

[0008] In the formula, s is a scale factor; K is the intrinsic matrix of the camera; R is the rotation matrix;t is a translation vector; u i , v i is a 2D coordinate of any feature point in the real-time image; P w is a 3D coordinate of the feature point in the world coordinate system.

[0009] As a further improvement of the present application, the specific operation of triangular meshing is as follows: extracting the outer wall geometry and structure information in the high-rise building BIM model; establishing a stable world coordinate system with the building key point as the origin in the model space. The outer wall surface and the key monitoring area are triangular meshed to obtain a discrete triangular mesh set.

[0010] As a further improvement of the present application, the way to obtain the number of pre-deployed gimbals and the pre-installed position of each gimbal is as follows: three-dimensional modeling is performed on the physical environment around the building where the gimbal needs to be deployed, and based on engineering specifications, visualization requirements and actual installation restrictions, a three-dimensional space candidate area where the gimbal can be deployed is defined, and the three-dimensional space candidate area is interval sampled to generate a series of pre-deployed positions of the gimbal.

[0011] As a further improvement of the present application, the process of obtaining the real-time image is as follows: instructing each gimbal to take pictures when it rotates in the horizontal direction and the pitch direction, and recording the time when the pictures are taken, thereby obtaining the real-time image.

[0012] As a further improvement of the present application, the process of obtaining the coordinate point pair corresponding to the successfully matched feature point in the real-time image is as follows: using an image feature matching algorithm to match feature points between the reference image and the real-time image, and using the matching result and the world coordinates of the feature points on the reference image to obtain the coordinate point pair corresponding to the successfully matched feature point set in the real-time image.

[0013] As a further improvement of the present application, the method of labeling feature points on the reference image is as follows: the labeling method of feature points on the reference image adopts a manual labeling method or an automatic labeling method; the feature points include window corners and wall decoration corner points on the reference image.

[0014] As a further improvement of the present application, the outer wall information of the building is divided into a plurality of triangular meshes after triangular meshing, and it can be judged whether any gimbal covers a certain triangular mesh through the visual coverage standard, and the judgment standard is as follows: when each influencing factor in the visual coverage standard meets the visual coverage standard, it means that the gimbal can cover the triangular mesh; wherein the influencing factors of the visual coverage standard include the field of view, the resolution, the self-shielding of the building and the external shielding of the building.

[0015] The application also comprises a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the gimbal camera deployment and automatic calibration method for high-rise building exterior wall fire positioning as described above when executing the computer program.

[0016] The application also comprises a storage medium, which is a computer-readable storage medium, and the computer program is stored on the storage medium, and the computer program implements the gimbal camera deployment and automatic calibration method for high-rise building exterior wall fire positioning as described above when executed.

[0017] The technical solution provided by the application has the following beneficial effects:

[0018] (1) The gimbal camera deployment and automatic calibration method for high-rise building exterior wall fire positioning provided by the application accurately guides the global optimization of the camera deployment position by using the BIM model, automatically establishes the association between the real-time image feature points and the BIM world coordinates by using the robust feature matching algorithm and PnP solution, and constructs the mapping relationship between the gimbal camera attitude parameter and the external parameter. This enables the accurate external parameter of the current camera to be obtained without re-shooting the calibration image in any real-time monitoring state (when the gimbal is tracking / scanning), so that the fire point pixel coordinates detected in the real-time image can be automatically and high-precision converted to the BIM world coordinate system, and the three-dimensional accurate positioning of the fire point on the building facade (to the floor and position) is realized, thereby providing key information for rapid emergency response. The core problems of unreasonable camera deployment, difficult automatic calibration, low positioning accuracy and dependence on manual work in high-rise building exterior wall fire monitoring are effectively solved, and the coverage, accuracy, automation degree and engineering practical value of the monitoring system are significantly improved.

[0019] (2) The gimbal camera deployment and automatic calibration method for high-rise building exterior wall fire positioning provided by the application covers the whole process from BIM data import, optimal deployment calculation, camera position and attitude configuration, calibration image acquisition, feature association matching to external parameter solving and updating. The manual measurement and manual calibration object layout links are minimized, the system deployment efficiency is greatly improved, the long-term maintenance cost is reduced, and the method is particularly suitable for intelligent monitoring of large-scale high-rise building groups.

[0020] (3) The gimbal camera deployment and automatic calibration method for high-rise building exterior wall fire positioning provided by the application can automatically solve the gimbal camera deployment scheme with the least number of cameras (or the lowest cost) or the highest coverage rate of the key area of the exterior wall under complex physical constraints by deeply mining the geometric information of the BIM model, accurately modeling the environmental obstacles, discretizing the camera parameter space, and combining advanced heuristic optimization algorithms, thereby significantly improving the coverage efficiency of the monitoring system and reducing the hardware investment cost and potential blind area. Attached Figure Description

[0021] Figure 1 This is a flowchart of the steps of the pan-tilt camera deployment and automatic calibration method for fire positioning on the exterior wall of high-rise buildings provided in Embodiment 1 of the present invention.

[0022] Figure 2 This is a schematic diagram of establishing a world coordinate system and performing triangulation on the exterior wall of a high-rise building in Embodiment 1 of the present invention.

[0023] Figure 3 This is a schematic diagram of the resolution of the exterior wall of a high-rise building in the Y-axis direction in Embodiment 1 of the present invention.

[0024] Figure 4 This is a schematic diagram of the reference image of the exterior wall of a high-rise building labeled with feature points in Embodiment 1 of the present invention.

[0025] Figure 5 This is a schematic diagram illustrating the matching of a reference image and a real-time image in Embodiment 1 of the present invention.

[0026] Figure 6 This is a schematic diagram of the PnP algorithm solution principle in Embodiment 1 of the present invention. Detailed Implementation

[0027] The present invention will now be further described in conjunction with specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0028] In the description of this invention, it should be noted that directional terms such as "center," "lateral," "longitudinal," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. They should not be construed as limiting the specific scope of protection of this invention. The terms "first," "second," etc., in the specification and claims of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The terms "comprising" and "having," and any variations thereof, in the specification and claims of this invention, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0029] Embodiment 1

[0030] The embodiment provides a gimbal camera deployment and automatic calibration method for high-rise building outer wall fire positioning. The method realizes deployment of an intelligent monitoring system with maximized coverage and automatic calibration process by combining a building information model (BIM), computational geometry, an optimization algorithm and computer vision technology. The BIM model is used to accurately guide global optimization of the camera deployment position, and model information is used to realize automatic calibration of multiple cameras at any position (that meets the deployment area constraint) without relying on a traditional cumbersome calibration board and manual intervention. As shown in Figure 1 , the embodiment specifically comprises the following steps:

[0031] (1) A world coordinate system is established according to the information of the high-rise building outer wall where the gimbal camera needs to be deployed, and the building outer wall is triangulated

[0032] The specific operation is as follows: in the embodiment, the BIM model can be used to extract the geometric and structural information of the outer wall of the high-rise building. A stable world coordinate system is established in the model space with the key points of the building (such as the center of the foundation or the specified corner points) as the origin. The outer wall surface and any area that needs to be monitored (such as the flammable insulation layer area) are finely triangulated to obtain a discrete triangular mesh set. These triangular meshes can be used as basic units for coverage evaluation and feature point association.

[0033] It can be understood that in other embodiments, the point cloud model of the high-rise building can also be collected by using an unmanned aerial laser radar camera in step (1), and the world coordinate system is established and triangulated to obtain Figure 2 .

[0034] (2) The number of gimbal cameras that can be deployed in the area around the outer wall of the high-rise building to be deployed and the pre-installation position of each gimbal camera are obtained

[0035] The specific operation is as follows: the physical environment (such as adjacent buildings, trees, street lamps, billboards, etc.) around the high-rise building is modeled in three dimensions. The three-dimensional modeling method can use a simple cuboid model or other more detailed models. Based on engineering specifications (such as wall attachment points, surrounding platforms), visibility requirements and actual installation restrictions, the three-dimensional candidate area (such as a parallel plane at a certain distance from the outer wall, the opposite roof area) where the gimbal camera can be deployed is defined. The three-dimensional candidate area where the gimbal camera can be deployed is sampled at intervals to generate the number of gimbal camera pre-deployments and the pre-installation position of each gimbal camera.

[0036] (3) The optimal deployment number of the gimbal camera and the optimal deployment position of each gimbal camera are obtained

[0037] In the embodiment, the actual number of gimbal cameras can be set in sequence according to the number of gimbal cameras pre-installed in step (ii), and then the maximum coverage rate and the corresponding position coordinates under the number of gimbal cameras are obtained by using the heuristic optimization algorithm. The number of gimbal cameras corresponding to the maximum coverage rate is taken as the optimal deployment number of gimbal cameras, and the position coordinates corresponding to each gimbal camera under the number are taken as the optimal deployment position of each gimbal camera.

[0038] The following can be described by the following example: if the maximum number of gimbal cameras pre-installed is 10, 10 groups of experimental groups are set, and the deployment number of gimbal cameras in the 10 groups of experimental groups is 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10, respectively. The heuristic optimization algorithm is used to obtain the maximum coverage rate of the gimbal cameras under the number and the position coordinates corresponding to each gimbal camera in each group under the maximum coverage rate. Then, the group with the maximum coverage rate is selected from the experimental groups, the number of gimbal cameras in the group is taken as the optimal deployment number of gimbal cameras, and the position coordinates corresponding to each gimbal camera under the number are taken as the optimal deployment position of each gimbal camera, thereby obtaining the optimal deployment number of gimbal cameras and the optimal deployment position of each gimbal camera.

[0039] The heuristic optimization algorithm can use genetic algorithm (GA), particle swarm optimization (PSO), and simulated annealing (SA). Since these optimization algorithms are common in the prior art, the specific optimization process by using the above optimization algorithms will not be described in detail in the present scheme.

[0040] In actual application, the cost of the gimbal camera can also be considered as a factor when designing the optimal deployment number of the gimbal camera. That is, when actually setting the deployment number of the gimbal camera, the cost of the gimbal camera and the coverage rate of the gimbal camera under different numbers are considered comprehensively to set the optimal deployment number of the gimbal camera. For example, if the number of gimbal cameras is 6, the coverage rate is 96%, and the coverage rate of 7 gimbal cameras is 97%, but the overall deployment cost will be greatly increased by adding one more gimbal camera. At this time, we can set the deployment number of the gimbal camera to 6 when considering the cost, thereby achieving a balance between high coverage and low cost of the outer wall monitoring.

[0041] Since the target building outer wall information monitored by the gimbal camera is divided into multiple triangular grids in the present application, when judging the coverage rate of the gimbal camera at each pre-deployment position, it is generally obtained by judging which specific triangular grids in the multiple triangular grids can be covered by each gimbal camera at the position. Therefore, in the present scheme, a visibility matrix can be constructed to represent whether a certain triangular grid is visible to a certain gimbal camera. It is assumed that there are n gimbal cameras and m triangular grids, and the visibility matrix V is constructed as follows:n Row m Visibility matrix of column VM to represent whether the triangular mesh is visible, as follows:

[0042] .

[0043] In the matrix, i represents the i th pan-tilt camera, i =1,2,…, n ; j represents the j th triangular mesh, j =1,2,…, m . M j covbyC i represents that the j th triangular mesh is covered by the i th pan-tilt camera, i.e., the triangular mesh is visible; M j uncovbyC i represents that the j th triangular mesh is not covered by the i th pan-tilt camera, i.e., the triangular mesh is invisible. Through the above matrix, the coverage rate of the outer wall of the high-rise building under different numbers of pan-tilt cameras can be obtained.

[0044] Specifically, for whether any pan-tilt camera is visible to any triangular mesh, a visual coverage standard can be used for judgment. The influencing factors of the visual coverage standard can include a field of view (FOV), a resolution, self-occlusion of the high-rise building, and external occlusion of the high-rise building. If the above four influencing factors all meet their own set requirements, it means that the pan-tilt camera is visible to the triangular mesh. If any one of the above four influencing factors does not meet its own set requirement, it means that the pan-tilt camera is invisible to the triangular mesh.

[0045] In the embodiment, the intrinsic parameters of the pan-tilt camera include: a maximum focal length f∈R + , in millimeters. The resolution: image width w∈N + and image height h∈N + , in pixels. The pixel size: horizontal s u ∈R + and vertical s v ∈R + ​​​​​​, in millimeter per pixel. The horizontal rotation angle and the vertical rotation angle of the PTZ camera.

[0046] For the field of view (FOV), let x ci , y ci , z ci represent the field of view of the PTZ camera c i The coordinates of the center of the lower triangular grid. α l , α r , α t , α b represent the left view angle, the right view angle, the upper view angle, and the lower view angle of the PTZ camera, respectively. Then the judgment criterion of the field of view can be as follows:

[0047] .

[0048] When the duration of the field of view of the PTZ camera satisfies the above formula, it means that the field of view of the PTZ camera meets the criterion. Otherwise, it means that the field of view of the PTZ camera does not meet the requirement.

[0049] For the resolution, the resolution is represented by the number of pixel points on the image plane per unit length of the object surface. Please refer to Figure 3 . Figure 3 The resolution of the outer wall of the high-rise building in the Y-axis direction is shown. Among them r ( P ci ) represents the resolution, θ i is the angle between the opposite direction of the normal vector of the triangular grid and the Z-axis. According to the principle of triangle similarity, the following formula can be obtained:

[0050] ;

[0051] In the formula, f is the focal length of the PTZ camera.

[0052] In the present scheme, let r a represent the minimum resolution required by the triangular grid, and only when the minimum resolution required in the X and Y directions is met, the resolution of the PTZ camera meets the constraint, at which time the resolution of the PTZ camera meets the visual coverage criterion. That is, when r ( P ci )≥ r aThe resolution of the PTZ camera meets the visual coverage standard at this time. In the present scheme r P ci The calculation formula of the focal length of the PTZ camera is as follows:

[0053]

[0054] In the formula, f is the focal length of the PTZ camera, f θ i is the angle between the opposite direction of the normal vector of the triangular mesh and the Z-axis; z ci is the Z-axis coordinate of the center of the triangular mesh; s u is the horizontal size of the pixel; s v is the vertical size of the pixel.

[0055] In the fire monitoring system of the high-rise building outer wall, the minimum resolution should be obtained according to the total pixels of the image, the PTZ camera parameters, the monitoring distance, and the detection algorithm used and the size of the flame that can be detected.

[0056] For the self-occlusion of the high-rise building, the judgment standard is as follows: if the line segment connecting the vertex of the triangular mesh and the PTZ camera intersects with the triangular mesh at an inner point of the triangular mesh, the vertex is occluded in the view of the PTZ camera. If one vertex in the triangular mesh is occluded, it indicates that the triangular mesh is occluded, that is, the self-occlusion of the high-rise building does not meet the visual coverage standard.

[0057] For the external occlusion of the high-rise building, the judgment standard is as follows: the obstacles around the high-rise building are modeled as cuboids, and the ray AABB bounding box detection method is used for detection. If the ray of the PTZ camera pre-triangular mesh reaches the AABB bounding box, it indicates that the triangular mesh is occluded, that is, the external occlusion of the high-rise building does not meet the visual coverage standard.

[0058] In the present scheme, by deeply mining several information of the BIM model, the obstacles in the environment around the building are accurately modeled, the parameter space of the PTZ camera is discretized, and combined with the heuristic optimization algorithm, the PTZ camera deployment scheme with the least number of cameras or the lowest cost or the highest coverage rate of the key area of the outer wall can be automatically solved under complex physical constraints, which significantly improves the coverage efficiency of the fire monitoring system and reduces the hardware investment cost and potential blind area.

[0059] (4) Obtain the reference image and the real-time image

[0060] ​​​The image of the building facade with the largest orientation taken by the gimbal camera at the optimal deployment position obtained in step (three) is used as the reference image, and real-time images of each gimbal camera at different poses are obtained.

[0061] The pose of the gimbal camera is adjusted by changing the pose parameters, and the pose parameters (P, T) include rotation in the horizontal direction and rotation in the vertical direction. P represents the rotation angle of the gimbal camera in the horizontal direction, and T represents the rotation angle of the gimbal camera in the vertical direction. The real-time images are obtained by the following operations: specifying each gimbal camera to rotate in the horizontal (Pan) and vertical (Tilt) directions within a certain range, while waiting for natural light / weather changes and recording the shooting time to take a series of real-time images. These images can represent the same building facade seen by the gimbal camera under different viewing angles, different scene conditions, and different times. The rotation range of the gimbal camera in the horizontal and vertical directions is less than the full range determined during coverage optimization.

[0062] (five) Label feature points and obtain coordinate point pairs

[0063] Please refer to Figure 4 Label feature points on the reference image and obtain 2D coordinates of the feature points on the reference image. Then, map the 2D coordinates of each feature point to the world coordinate system through the BIM model to obtain the 3D coordinates of the feature points in the world coordinate system. Then, construct the coordinate point pairs of the 2D coordinates and 3D coordinates of each feature point.

[0064] In this embodiment, SIFT / SURF / ORB and other automatic labeling methods can be used to extract image feature points, or LabeMe tools can be used to manually label feature points. Then, the BIM model in step (one) and the known deployment position of the gimbal camera are used to obtain the coordinate point pairs. The feature points can be selected from positions with high appearance differentiation and relatively stable to viewing angle changes, such as window corners and decorative corner points.

[0065] (six) Use image feature matching algorithm to match feature points between reference image and real-time image

[0066] Please refer to Figure 5, and the world coordinates of the feature points on the known reference image, to obtain the coordinate point pair corresponding to the matched feature point set in the real-time image. The image feature matching algorithm can be SuperGlue, XFeat, or traditional SIFT / SURF / ORB+RANSAC based on deep learning. For XFeat, XFeat is a novel lightweight CNN architecture that can be deployed on resource-constrained platforms and downstream human ways that require high throughput or computational efficiency without time-consuming hardware feature optimization. XFeat can easily replace existing lightweight handcrafted solutions, expensive deep models, and lightweight deep models in downstream tasks such as visual positioning and camera pose estimation. A novel matching refinement module is proposed to obtain pixel-level offsets from coarse semi-dense matching. Compared with the prior art, XFeat does not require high-resolution features in addition to local description, greatly reducing the amount of calculation.

[0067] (Seven) The mapping relationship between the coordinate point pair and the extrinsic parameters of the gimbal camera is constructed by the PnP algorithm.

[0068] In the present scheme, the mapping relationship between the coordinate point pair of the feature point and the extrinsic parameters of the gimbal camera is as follows:

[0069] ;

[0070] In the formula, s is a scale factor, which is used for homogeneous coordinates; K is the intrinsic parameter matrix of the camera; R is the rotation matrix; t is the translation vector; u i , v i is the 2D coordinate of any feature point in the real-time image; P w is the 3D coordinate of the feature point in the world coordinate system.

[0071] In this step, the extrinsic parameters of the gimbal camera include the rotation matrix R and the translation vector tPnP (Perspective-n-Point) algorithm, which refers to calculating the projection relationship between a plurality of feature points in the world coordinate and a plurality of image points in the image imaging thereof, so as to obtain the external parameter of the PTZ camera. The PnP algorithm can include EPnP algorithm, DLT algorithm, UPnP algorithm or robust solver, etc. The robust solver can be RANSAC + iterative method of minimizing re-projection error. And there are three methods for estimating the external parameter of the camera: (1) 2D-2D method, which estimates the external parameter of the camera based on at least 8 pairs of 2D feature points and their epipolar constraints. (2) 3D-3D method, which estimates the external parameter of the camera based on the iterative closest point (ICP) algorithm by matching two groups of 3D feature points. (3) 3D-2D method, which estimates the external parameter of the camera by solving the PnP problem when the 3D points and their 2D projections on the image are known. Compared with the 2D-2D method and the 3D-3D method, the PnP method does not need to use the epipolar constraint, and can obtain better camera external parameter estimation results in the case of a small number of matching points. The PnP algorithm principle diagram is shown in Figure 6 . The following describes the PnP algorithm.

[0072] Known: n 3D points P w ( X w , Y w , Z w ) in the world coordinate system, their corresponding 2D pixel points in the image are p i u i , v i . For each real-time image, the coordinate point pair obtained in step (five) and the internal parameter matrix of the PTZ camera which is pre-calibrated or has been calibrated K are used, and the PnP algorithm is applied to solve the rigid body transformation (i.e. the rotation matrix R and the translation vector t ) of the PTZ camera relative to the world coordinate system at the moment of shooting the real-time image. Wherein, the number of coordinate point pairs needs to be selected to be greater than or equal to 4 groups. The present scheme significantly improves the robustness of feature matching and external parameter solving by using the stable and rich feature sources (such as specially designed window corners and structural parts) provided by the BIM model instead of sparse or variable features extracted on site, and combining multi-view images for statistical optimization, which adapts to different light, weather and view changes.

[0073] ​(Eight) based on the real-time image of the PTZ camera in different postures to obtain multiple sets of extrinsic parameter data, and then construct the mapping relationship between the posture parameters (P, T) and the extrinsic parameters (R, t) of the PTZ camera. Then, through the above mapping relationship, the fire point pixel coordinates detected in the real-time image in the determined posture are converted into world coordinates, and then the positioning of the fire point on the external wall is realized.

[0074] It can be understood that, in the present scheme, through the mapping relationship between the posture parameters and the extrinsic parameters of the PTZ camera, when the PTZ camera is rotated to a new position, the system can obtain or estimate the extrinsic parameters of the PTZ camera in the posture according to the above mapping relationship, and then through the mapping relationship between the coordinate point pair of the feature point and the extrinsic parameters of the PTZ camera, the fire point pixel coordinates detected in the real-time image can be converted into world coordinates, and then the positioning of the fire point on the external wall is realized.

[0075] The present application constructs the mapping relationship between the posture parameters and the extrinsic parameters of the PTZ camera through the automatic calibration of the PTZ camera, so that the fire monitoring and positioning system can obtain the accurate extrinsic parameters of the PTZ camera in any real-time monitoring state after using the above calibration method of the PTZ camera, and then through the mapping relationship between the posture parameters and the extrinsic parameters and the mapping relationship between the extrinsic parameters and the coordinate point pair, the fire point pixel coordinates detected in the real-time image can be automatically and high-precisionly converted into the BIM world coordinate system, so as to realize the three-dimensional accurate positioning of the fire point on the building facade, which can be specific to the floor and the position on the floor, thereby providing key information for the rapid emergency response to the fire. It can effectively solve the technical problems of the traditional camera calibration method (such as using a fixed calibration board) which depends on the fixed relative position of the camera and the calibration board, and cannot adapt to the dynamic characteristics of the PTZ camera (such as the original extrinsic parameters being invalid after the PTZ camera is rotated or zoomed, and if recalibration depends on manual repositioning of the calibration board or complex algorithms). Moreover, the dynamic automatic calibration method of the present scheme can also solve the problem that the calibration board in the existing camera calibration method is prone to cause positioning accuracy of real-time shooting images to be unguaranteed due to difficulty in operation, extremely low efficiency or inability to complete real-time calibration in high-rise building scenarios.

[0076] The present application innovatively uses BIM as a high-precision, prior calibration source, and combines the controllable motion characteristics of the PTZ camera, so as to actively collect multi-view / multi-scene images by driving the PTZ camera through instructions, and then automatically establish the mapping relationship between the feature points in the real-time image and the BIM world coordinates by using robust feature matching algorithms and PnP solving. Finally, by constructing the mapping relationship between the posture parameters and the extrinsic parameters of the PTZ camera, the PTZ camera after calibration can obtain accurate extrinsic parameter estimation without the need for re-shooting calibration images in any posture within the mapping range, and then realize the accurate positioning of the three-dimensional coordinates of the fire point on the building facade.

[0077] The application provides a gimbal camera deployment and automatic calibration method for high-rise building outer wall fire positioning, which covers the whole process from BIM data import, optimal deployment quantity and optimal deployment position of each gimbal camera, gimbal camera posture configuration, acquisition of calibration images, mapping relationship between coordinate points of feature points and external parameters, and mapping relationship between external parameters and posture parameters of the gimbal camera. The whole process is based on the optimized deployment of BIM and dynamic automatic calibration based on BIM and multiple mapping relationships, so as to form a complete high-rise building outer wall fire monitoring pre-deployment and actual pose perception support system. The whole process is highly automated, which can minimize manual measurement, manual calibration and other links, greatly improve the deployment efficiency of the fire monitoring system, reduce the long-term maintenance cost, and is particularly suitable for intelligent monitoring of large-scale high-rise building groups.

[0078] Embodiment 2

[0079] The embodiment provides a gimbal camera deployment and automatic calibration method for high-rise building outer wall fire positioning in a specific scene based on the embodiment 1, which comprises the following steps:

[0080] I. The outer wall geometry and structure information of the high-rise building is extracted through the point cloud model. A stable world coordinate system with the key point (such as the center of the foundation or the specified corner point) of the building as the origin is established in the model space. The outer wall surface and any area that needs to be monitored (such as the flammable insulation layer area) are finely triangulated to obtain a discrete triangular mesh set. In this embodiment, 20000 triangular meshes are obtained after the above processing.

[0081] II. The physical environment (such as adjacent buildings, trees, street lamps, billboards, etc.) around the high-rise building is modeled in three dimensions. The three-dimensional modeling method can use a simple cuboid model or other more detailed models. Based on engineering specifications (such as wall attachment points, surrounding platforms), visibility requirements and actual installation restrictions, the three-dimensional candidate area (such as parallel planes at a certain distance from the outer wall, opposite roof areas) where the gimbal camera can be deployed is defined. The three-dimensional candidate area where the gimbal camera can be deployed is sampled at intervals to generate the number of gimbal camera pre-deployments and the pre-installation position of each gimbal camera.

[0082] In this embodiment, the parameters of the gimbal camera are shown in the following table:

[0083] Table 1 is the parameter table of the gimbal camera in this embodiment

[0084]

[0085] And the pre-deployed position of the gimbal camera obtained after step II is 637.

[0086] Three, set the actual number of gimbal cameras in turn according to the number of gimbal cameras pre-installed in step (two) of the cloud, and obtain the maximum coverage and the corresponding position coordinates under the number through the heuristic optimization algorithm. The number of gimbal cameras corresponding to the maximum coverage is taken as the optimal deployment number of gimbal cameras, and the position coordinates corresponding to each gimbal camera under the number are taken as the optimal deployment position of each gimbal camera. In this embodiment, the optimal deployment number, coverage and position of each gimbal camera obtained based on the specific data above are shown in Table 2.

[0087] Table 2 is a table of the optimal deployment number, coverage and optimal deployment position of each gimbal camera obtained by the heuristic optimization algorithm

[0088]

[0089] Four, use the image with the largest orientation of the building outer wall obtained by the gimbal camera in step (three) as the reference image, and obtain the real-time image of each gimbal camera under different postures.

[0090] Five, label the feature points on the reference image, obtain the 2D coordinates of the feature points on the reference image, and then map the 2D coordinates of each feature point to the world coordinate system through the point cloud model to obtain the 3D coordinates of the feature points in the world coordinate system; and further construct the coordinate point pairs of the 2D coordinates and 3D coordinates of each feature point.

[0091] Six, use the image feature matching algorithm to match the feature points between the reference image and the real-time image. Use the matching results and the world coordinates of the feature points on the reference image to obtain the coordinate point pairs corresponding to the matched feature point set on the real-time image.

[0092] In this embodiment, the coordinate point pairs of the matched feature points between the real-time image and the reference image are shown in Table 3.

[0093] Table 3 is a table of the coordinate point pairs of the matched feature points between the real-time image and the reference image

[0094]

[0095] Seven, construct the mapping relationship between the coordinate point pairs and the extrinsic parameters of the gimbal camera through the PnP algorithm. The mapping relationship between the coordinate point pairs of the feature points and the extrinsic parameters of the gimbal camera is as follows:

[0096] ;

[0097] In the formula, s is a scale factor, which is used for homogeneous coordinates; K is the intrinsic parameter matrix of the camera; R is the rotation matrix;t is a translation vector; u i , v i is a 2D coordinate of any feature point in the real-time image; P w is a 3D coordinate of the feature point in the world coordinate system.

[0098] This embodiment deploys a gimbal camera at a site, captures a reference image and a real-time image, and performs a set of verifications. The intrinsic matrix of the gimbal camera is:

[0099] ;

[0100] PnP is used to solve, and the rotation matrix R is obtained after RANSAC optimization:

[0101] [[ 0.299727 -0.953505 0.031492]

[0102] [-0.058215 -0.051228 -0.996989]

[0103] [ 0.952247 0.296991 -0.070863]]

[0104] The translation vector t is [22.414495 58.793131 36.507256].

[0105] Eight, based on the real-time images of the gimbal camera in different poses, a plurality of sets of extrinsic parameter data are obtained, and a mapping relationship between the pose parameters and the extrinsic parameters of the gimbal camera is constructed. Then, through the above mapping relationship, the pixel coordinates of the fire points detected in the real-time image in the determined pose are converted into the world coordinates, and the positioning of the external wall fire points is realized. It can be understood that, in the present scheme, through the mapping relationship between the pose parameters and the extrinsic parameters of the gimbal camera, when the gimbal camera is rotated to a new position, the system can immediately obtain or estimate the extrinsic parameters of the gimbal camera in the above pose according to the above mapping relationship, and then through the mapping relationship between the coordinate points of the feature points and the extrinsic parameters of the gimbal camera, the pixel coordinates of the fire points detected in the real-time image can be converted into the world coordinates, and the positioning of the external wall fire points is realized.

[0106] Embodiment 3

[0107] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the gimbal camera deployment and automatic calibration method for high-rise building external wall fire positioning of embodiment 1 are implemented.

[0108] The computer device can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including a single server or a server cluster composed of multiple servers), etc. which can execute programs. The computer device of the embodiment includes at least but not limited to a memory and a processor which can be connected to each other for communication through a system bus.

[0109] In the embodiment, the memory (i.e. a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g. an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the memory can also include both the internal storage unit and the external storage device of the computer device. In the embodiment, the memory is usually used to store an operating system and various application software installed on the computer device, etc. In addition, the memory can also be used to temporarily store various data which have been output or will be output.

[0110] The processor in some embodiments can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor is usually used to control the overall operation of the computer device. In the embodiment, the processor is used to run the program codes or process data stored in the memory to implement the steps of the gimbal camera deployment and automatic calibration method for high-rise building external wall fire positioning in embodiment 1.

[0111] Embodiment 4

[0112] The embodiment discloses a readable storage medium, and the readable storage medium stores computer program instructions. When the computer program instructions are read and run by a processor, the steps of the gimbal camera deployment and automatic calibration method for high-rise building external wall fire positioning in embodiment 1 are executed.

[0113] The gimbal camera deployment and automatic calibration method for high-rise building outer wall fire positioning in embodiment 1 can be applied in the form of software, such as a program designed to be independently run on a computer readable storage medium, and the computer readable storage medium can be a U disk. The U disk is designed to start the entire method by an external trigger.

[0114] The above describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection required by the present application is defined by the appended claims and their equivalents.

Claims

1. A gimbal camera deployment and automatic calibration method for high-rise building exterior wall fire positioning, characterized in that, It comprises: Extracting building exterior wall information, establishing a world coordinate system, and triangulating the building exterior wall; Three-dimensional modeling of obstacles around the building to obtain the pre-deployment position of the camera; Setting the number of deployed cameras, obtaining the maximum coverage rate of the cameras under this number and its corresponding position coordinates through heuristic optimization algorithm; the maximum coverage rate of the camera number and position coordinates are the optimal deployment number and optimal deployment position, Setting the reference image of the camera, and obtaining the real-time image of the camera under different postures; Labeling feature points on the reference image and obtaining the 2D coordinates of the feature points; then constructing the 2D coordinates and 3D coordinates of the feature points according to the BIM model and the world coordinate system; Matching the reference image and the real-time image and obtaining the coordinate point pair corresponding to the matched feature point set in the real-time image; constructing the mapping relationship between the coordinate point pair and the camera's external parameter; Based on the multiple sets of external parameter data obtained from the real-time image, the mapping relationship between the camera's posture parameter and the external parameter is constructed; through the above mapping relationship, the pixel coordinates of the fire point detected in the real-time image under the determined posture are converted into the world coordinate system to realize the positioning of the exterior wall fire point.

2. The PTZ camera deployment and automatic calibration method for high-rise building exterior wall fire positioning of claim 1, wherein, The external parameter of the gimbal camera includes a rotation matrix R and a translation vector t ; a 2D coordinate of any image point in a real-time image captured by the gimbal camera is defined as p i = (x, y) u i , v i , and a corresponding 3D coordinate of the image point in a world coordinate system is P w = (X, Y, Z) X w , Y w , Z w ; a mapping relationship between the coordinate point pair of the feature point and the external parameter of the gimbal camera is as follows: ； where s is a scale factor; K is the intrinsic matrix of the camera; R is the rotation matrix; t is the translation vector; u i , v i is the 2D coordinate of any feature point in the real-time image; P w is the 3D coordinate of the feature point in the world coordinate system.

3. The PTZ camera deployment and automatic calibration method for high-rise building exterior wall fire positioning of claim 1, wherein, The specific operation of triangulation is as follows: extracting the exterior wall geometry and structure information from the high-rise building BIM model; establishing a stable world coordinate system with the building key point as the origin in the model space; triangulating the exterior wall surface and the key monitoring area to obtain a discrete triangular mesh set.

4. The PTZ camera deployment and automatic calibration method for high-rise building exterior wall fire positioning of claim 1, wherein, The way to obtain the pre-deployment number of the pan-tilt camera and the pre-installation position of each pan-tilt camera is as follows: three-dimensional modeling of the physical environment around the building where the pan-tilt camera needs to be deployed, and based on engineering specifications, visualization requirements and actual installation restrictions to define the three-dimensional space candidate area where the pan-tilt camera can be deployed, interval sampling of the three-dimensional space candidate area to generate a series of pre-deployment positions of the pan-tilt camera.

5. The PTZ camera deployment and automatic calibration method for high-rise building exterior wall fire positioning of claim 1, wherein, The process of obtaining real-time images is as follows: instructing each pan-tilt camera to shoot images when the horizontal direction and the pitch direction of the camera rotate, and recording the time when the images are shot, and then obtaining the real-time images.

6. The PTZ camera deployment and automatic calibration method for high-rise building exterior wall fire positioning of claim 1, wherein, The process of obtaining the coordinate point pair corresponding to the matched feature point in the real-time image is as follows: using image feature matching algorithm to match feature points between the reference image and the real-time image, and using the matching result and the known world coordinates of the feature points on the reference image to obtain the coordinate point pair corresponding to the matched feature point set in the real-time image.

7. The PTZ camera deployment and automatic calibration method for high-rise building exterior wall fire positioning of claim 1, wherein, The method of labeling feature points on the reference image adopts manual labeling method or automatic labeling method for labeling; the feature points include window corners and wall decoration corner points on the reference image.

8. The PTZ camera deployment and automatic calibration method for high-rise building exterior wall fire positioning of claim 1, wherein, After the exterior wall information of the building is triangulated, it is divided into multiple triangular meshes, and it can be judged whether any pan-tilt camera covers a certain triangular mesh through the visual coverage standard, and the judgment standard is as follows: when each factor in the visual coverage standard meets the visual coverage standard, it means that the pan-tilt camera can cover the triangular mesh; wherein the influence factors of the visual coverage standard include field of view, resolution, self-shielding of the building and external shielding of the building.

9. A computer device, comprising: The computer program product comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the gimbal camera deployment and automatic calibration method for high-rise building external wall fire positioning according to any one of claims 1-8 when executing the computer program.

10. A storage medium, characterized by The computer program product comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the gimbal camera deployment and automatic calibration method for high-rise building external wall fire positioning according to any one of claims 1-8 when executing the computer program.

Images