Fire monitoring method based on monocular camera and digital surface model

Abstract

The invention provides a fire monitoring method based on a monocular camera and a digital surface model, and the method comprises the steps: carrying out the calibration and correction of a camera through a Zhang Zhengyou calibration method, and eliminating the impact of lens distortion; constructing a fire point target detection sample data set, and carrying out model training, verification, testing and deployment by adopting YOLOv5 to realize fire point identification and return a prediction frame coordinate; calculating pixel coordinates and image coordinates of the target, and calculating an azimuth angle and a pitch angle from the camera to the target by combining the azimuth angle and the pitch angle of the camera; and finally, acquiring latitude and longitude and elevation information of pixel points on the azimuth angle of the camera, calculating the distance, the elevation difference and the pitch angle, and comparing to obtain the actual coordinates of the target. The method effectively avoids the limitation of a traditional distance measurement mode, is wide in application scene and low in hardware cost, and can quickly and accurately carry out fire point positioning.

Description

Technical Field

[0001] The present invention relates to the technical field of fire point recognition and positioning, and particularly to a fire monitoring method based on a monocular camera and a digital surface model. Background Technique

[0002] Monocular camera positioning is an important branch of target positioning technology, mainly divided into the following two research directions: 1. Target detection and positioning methods based on traditional monocular positioning models. This method uses the principle of similar triangles to estimate the depth of a target with a known actual size; and obtains target box information through a target detection algorithm to achieve target positioning. In actual application scenarios, it is difficult to obtain the actual size of the target, so the target depth information is inaccurate, bringing positioning errors to target positioning. 2. Target positioning methods based on deep learning. An end-to-end deep learning method is used to detect the target, obtain the depth information of the target, and then combine the target detection algorithm for positioning. Such methods require the depth information of the target as the label of the data set, and the stability of the model is poor in an unfamiliar environment. In camera positioning technology, in addition to monocular ranging and positioning, binocular ranging, laser ranging, etc. are also commonly used for distance measurement of targets. Although binocular ranging can estimate the distance by simulating the principle of human binocular stereo vision, its accuracy and applicable range are limited by the baseline length between cameras, the target texture complexity, and the complexity; although laser ranging is direct and accurate, it may be severely interfered in harsh environments (such as a smoky fire scene) and is difficult to apply to target recognition and positioning of cameras in actual scenarios.

[0003] In view of this, we propose an innovative fire monitoring method that combines the high flexibility of a monocular camera and the accurate spatial information of a digital surface model, aiming to achieve an accurate estimation of the fire point target. Specifically, in implementation, the fire point position is identified through target monitoring technology, and combined with the camera pose and the digital surface model of this area, the accurate position of the fire point is estimated through mathematical relationships. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a fire monitoring method based on a monocular camera and a digital surface model, and to achieve rapid fire monitoring through the monocular camera and the digital surface model.

[0005] To solve the above technical problem, the technical solution adopted by the present invention is: a fire monitoring method based on a monocular camera and a digital surface model, including the following steps: Step 1, camera calibration and camera correction; Step 2, fire point recognition; Step 3, calculate the azimuth angle and elevation angle of the target; Step 4, calculate the actual coordinates of the target.

[0006] Preferably, in step 1, a calibration object with known dimensions is used. Through the correspondence between the points with known coordinates on the calibration object and their image points, Zhang Zhengyou calibration method is used for camera calibration to obtain the internal and external parameters of the camera model, solve the distortion parameters, and perform camera correction to eliminate the influence of lens distortion on the image quality.

[0007] Preferably, in step 2, fire point detection is carried out through a deep learning object detection model. A fire point target detection sample data set is constructed, and YOLOv5 is used for model training, verification, testing and deployment; the image captured by the camera is sent into the trained object detection network to judge whether there is a fire point and return the predicted box coordinates of the fire point.

[0008] Preferably, after completing fire point recognition and obtaining the predicted box of the fire point in step 3, calculate the pixel coordinates and image coordinates of the target, and use the mathematical function relationship based on the azimuth angle and pitch angle of the camera to calculate the azimuth angle and pitch angle from the camera to the target.

[0009] Preferably, in step 4, obtain the longitude, latitude coordinates and elevation information of all pixel points of the camera in the azimuth angle, calculate the distance and elevation difference between each pixel point and the camera position, deduce the pitch angle from the camera to each pixel point, compare it with the pitch angle from the camera to the target, and then obtain the actual coordinates of the target.

[0010] Preferably, the camera calibration and camera correction are specifically as follows: 1) Camera calibration: Prepare a two-dimensional checkerboard calibration board, and use the camera to take pictures of the calibration object from multiple angles. Use Zhang Zhengyou calibration method for camera calibration to solve the internal parameter matrix, external parameter matrix and distortion parameters of the camera. The internal parameter matrix includes pixel focal length , and the pixel coordinates of the principal point ; 2) Camera correction: Based on the parameters obtained by camera calibration, apply the distortion correction algorithm to correct the image.

[0011] Preferably, the fire point recognition is specifically as follows: 1) Target detection model training and optimization: Prepare a target detection fire point data set and perform data cleaning. Convert the data set format to the yolo format and divide the training set and validation set according to a certain ratio; use the YOLOv5 target detection network for model training, and adjust the hyperparameters of model training in a timely manner to optimize the training results; 2) Model testing and deployment: After completing the training of the target detection model, test it on an independent test set to evaluate the model performance, and deploy it for actual applications to achieve monocular camera fire point recognition and obtain the predicted box position of the fire point in the image.

[0012] Preferably, the calculation of the target azimuth angle and pitch angle is specifically as follows: 1) Calculate the pitch angles of all pixel points on the digital surface model in the direction of the camera position and the target azimuth angle. In three-dimensional space, denote the longitude, latitude, and elevation of the camera as 、 、 , taking the camera as the origin, obtain the longitude, latitude of all pixel points of the camera on the digital elevation model image at the azimuth angle α 0, and the elevation of the digital surface model, denoted as ; for all pixel points, calculate the horizontal distance between the camera coordinates and the pixel point coordinates and the elevation difference in the vertical direction; 2) Calculate the actual coordinates of the target. Successively find the angles α between each pixel point in the direction of the camera azimuth angle and the vertical direction between each pixel point and the camera . If the angles calculated for two pixel points are on both sides of the pitch angle from the camera to the target, then the average value of the longitudes and latitudes of the corresponding two pixel points is the actual coordinates of the target sought.

[0013] Preferably, calculate the horizontal distance between the camera coordinates and the pixel point coordinates : ; ; ; where 、 are the longitude and latitude of the camera respectively, 、 are the longitude and latitude of the pixel point coordinates respectively; is the distance between two points, is the average radius of the earth; Calculate the elevation difference between the camera coordinates and the pixel point coordinates in the vertical direction: ; where is the elevation of the pixel point, is the camera elevation.

[0014] Calculate the camera coordinates and pixel coordinates pitch angle : .

[0015] The present invention provides a fire monitoring method based on a monocular camera and a digital surface model, which has the following beneficial effects: 1. Fast and accurate fire point positioning, wide application scenarios: This method combines a monocular camera and a digital surface model. Through a series of steps such as camera calibration and correction, fire point recognition, calculation of azimuth angle and pitch angle, and actual coordinates, it can quickly and accurately locate the fire point. Compared with traditional methods such as laser ranging and binocular ranging, it gets rid of the dependence on special hardware conditions, overcomes the problems of laser ranging being easily interfered in harsh environments and the limited accuracy and application range of binocular ranging, and can be applied in more complex environments, such as a smoky fire scene, greatly broadening the application scenarios of fire monitoring.

[0016] 2. Low hardware cost: It can realize the recognition and positioning of fire points only relying on a monocular camera, a pan-tilt head, and digital surface model data, without the need for complex and expensive equipment such as laser rangefinders and binocular ranging systems, reducing the investment in hardware costs. For scenarios of large-scale deployment of fire monitoring systems, it can effectively control costs and improve the feasibility and economy of the solution.

[0017] 3. Using deep learning to improve the fire point recognition ability: Adopting a deep learning object detection model (such as YOLOv5) for fire point detection, by constructing a sample data set and training and validating the model, it can improve the accuracy and stability of fire point recognition. Data augmentation techniques and hyperparameter tuning optimize the model performance, enabling the model to effectively identify fire points in different scenarios and reducing misjudgment and missed judgment situations.

[0018] 4. Data processing and calculation improve the positioning accuracy: Camera calibration and correction eliminate the influence of lens distortion on image quality, ensuring the accuracy of image data. When calculating the target azimuth angle, pitch angle, and actual coordinates, mathematical functions and digital surface model information are used, fully considering factors such as the camera position, pixel coordinates, and elevation, improving the accuracy of fire point positioning and providing strong support for taking timely and accurate fire extinguishing measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The following further illustrates the present invention in conjunction with the drawings and embodiments: Figure 1 is the method flow chart of the present invention; Figure 2 is the schematic diagram of the principle of obtaining the target image coordinates and azimuth angle in the present invention; Figure 3 is the schematic diagram of target positioning in the present invention. Detailed implementation mode

[0020] As Figure 1 shown, a fire monitoring method based on a monocular camera and a digital surface model includes the following steps: 1. Camera calibration and camera correction 2. Fire point recognition 3. Calculate the target azimuth angle and pitch angle 4. Calculate the actual coordinates of the target To implement the camera calibration and camera correction of this example, the method is as follows: 1-1) Calibration preparation. Prepare a camera and a planar checkerboard calibration board, place the calibration board within the camera's field of view, and capture calibration board images at multiple different angles and positions; 1-2) Corner detection and matching. Use a corner detection algorithm to extract the key points on the calibration board, match the corner points in the image with the actual corner points on the calibration board, and obtain their coordinates in the image; 1-3) Intrinsic and extrinsic parameter estimation and distortion parameter solution. Map the points in the world coordinate system to the image coordinate system, construct an overdetermined equation through multiple sets of corresponding points, and use the least squares method to solve the homography matrix. Based on the relationship between the homography matrix and the camera's intrinsic parameters, initially calculate the camera's intrinsic parameter matrix. On the basis of knowing the camera's intrinsic parameters, further calculate the camera's extrinsic parameters (rotation matrix and translation vector) corresponding to each image through the homography matrix; use the initially calculated intrinsic and extrinsic parameters as the true values, and solve the distortion parameters by minimizing the reprojection error.

[0021] 1-4) Camera distortion correction and optimization. After obtaining the distortion parameters, perform distortion correction on the original image to eliminate the influence of lens distortion on the image quality.

[0022] To implement the fire point recognition of this example, the method is as follows: 2-1) Dataset preparation. Prepare a labeled fire point dataset. First, perform data cleaning to ensure the accuracy and consistency of the dataset; subsequently, to improve the generalization ability of the model, apply data augmentation techniques such as random cropping, rotation, and scaling to enhance the diversity of training samples; then divide the dataset into a training set and a validation set according to a certain ratio, and create a configuration file (such as a.yaml file) according to the specific requirements of YOLOv5, specifying in detail parameters such as the dataset path, class names, image size, batch size, and learning rate decay strategy to prepare for subsequent training.

[0023] 2-2) Model training and tuning. Use the training script provided by the official YOLOv5 for training, and perform transfer learning by automatically loading pre-trained weights; during the training process, adjust the hyperparameters of the model in a timely manner according to the performance of the validation set, such as the learning rate, batch size, number of training epochs, and momentum, etc., to find the optimal training configuration; after each round of training, use the validation set to evaluate the model, record key indicators such as loss value and mAP, and monitor the training progress and performance of the model.

[0024] 2-3) Model testing and model deployment. Test the trained model on an independent test set to evaluate its recognition accuracy and generalization ability in actual applications; optimize the model for the target hardware platform, adjust the inference engine configuration, optimize memory management, etc., to ensure the efficient operation of the model in actual deployment; finally, deploy the optimized model to the actual application scenario, capture images in real time through a monocular camera, and use the trained model for fire point recognition, and return the predicted box coordinate information of the fire point.

[0025] As Figure 2 shown, to achieve the calculation of the target azimuth and elevation angles in this example, the method is as follows: 3-1) Obtain image coordinates After completing the distortion correction of the image and obtaining the predicted box of the fire point, mark the lower left corner of the image as the pixel coordinate origin, and the positive directions of the x-axis and y-axis are to the right and up respectively . The coordinates of the fire point predicted box in pixel coordinates are ; mark the center point of the image as the image coordinate origin, and the positive directions of the x-axis and y-axis are to the right and up respectively . Denote the image coordinates of the fire point predicted box as . The width and height of the camera image are respectively. Calculate the pixel coordinates of the fire point predicted box as : ; 3-2) Calculate the azimuth and elevation angles from the camera principal point to the target.

[0026] To determine the exact direction from the camera to the fire point, first, the current azimuth angle and elevation angle of the monocular camera, as well as the world coordinates of the camera (i.e., the position of the camera in three-dimensional space), need to be obtained. Based on the target image coordinates and the pixel focal lengths , in the camera intrinsic matrix, use trigonometric relationships to calculate the azimuth angle and elevation angle from the camera to the target: ; To achieve the calculation of the actual coordinates of the target in this example, the method is as follows: 4-1) Calculate the pitch angles of all pixel points on the digital surface model in the direction of the camera position and the target azimuth angle.

[0027] Figure 3 is a schematic diagram of target positioning, where the camera position is , and the three-dimensional coordinates of each pixel point in the direction of the camera to the target azimuth angle are . In the three-dimensional space, sequentially obtain the camera coordinates in the azimuth angle direction and the coordinates of each pixel point in the horizontal direction and the elevation difference in the vertical direction .

[0028] ; ; ; ; Among them, , are the longitude and latitude of the camera respectively, , are the longitude and latitude of the pixel point coordinates (in radians); is the distance between two points (the unit is usually kilometers), is the average radius of the earth (about 6371 kilometers); is the elevation of the pixel point, is the camera elevation.

[0029] Calculate the pitch angle of the camera coordinates and the pixel point coordinates :

[0030] 4-1) Calculate the actual coordinates of the target Sequentially find the angles between each pixel point in the direction of the camera azimuth angle and the camera in the vertical direction in the three-dimensional space. If the calculated angles of two pixel points are on both sides of the pitch angle

[0031] The present invention provides a fire monitoring method based on a monocular camera and a digital surface model. First, the Zhang-Zhengyou calibration method is used for camera calibration and correction to eliminate the influence of lens distortion. Then, by constructing a fire point target detection sample data set, YOLOv5 is used for model training, verification, testing and deployment to achieve fire point recognition and return the coordinates of the prediction box. Next, the pixel coordinates and image coordinates of the target are calculated, and the azimuth and elevation angles from the camera to the target are calculated in combination with the azimuth angle and elevation angle of the camera. Finally, the longitude, latitude and elevation information of the pixel points on the camera azimuth are obtained, the distance, elevation difference and elevation angle are calculated, and the actual coordinates of the target are obtained by comparison. This method effectively avoids the limitations of traditional ranging methods, has a wide range of application scenarios and low hardware costs, and can quickly and accurately locate fire points.

[0032] The above embodiments are only the preferred technical solutions of the present invention and should not be regarded as limitations to the present invention. The protection scope of the present invention should be the technical solutions recorded in the claims, including the equivalent replacement solutions of the technical features in the technical solutions recorded in the claims. That is, the equivalent replacement improvements within this scope are also within the protection scope of the present invention.

Claims

1. A fire monitoring method based on a monocular camera and a digital surface model, characterized in that: The following steps are involved: Step 1: Camera calibration and camera correction; Step 2: Fire point identification; Step 3, calculate the target azimuth and pitch angle; Step 4: Calculate the actual coordinates of the target.

2. A fire monitoring method based on a monocular camera and a digital surface model according to claim 1, characterized in that: The step 1 uses a calibration object of known size, and through the correspondence between points on the calibration object with known coordinates and their image points, uses the Zhang Zhengyou calibration method to calibrate the camera, obtain the internal and external parameters of the camera model, solve the distortion parameters, perform camera correction, and eliminate the influence of lens distortion on image quality.

3. The fire monitoring method based on a monocular camera and a digital surface model according to claim 1, characterized in that: The step 2 performs fire point detection through a deep learning target detection model, constructs a fire point target detection sample data set, and uses YOLOv5 to train, verify, test and deploy the model; the image taken by the camera is sent to the trained target detection network to determine whether there is a fire point and return the predicted frame coordinates of the fire point.

4. The fire monitoring method based on a monocular camera and a digital surface model according to claim 1, characterized in that: After the fire point identification is completed in step 3 to obtain the predicted frame of the fire point, the pixel coordinates and image coordinates of the target are calculated, and the azimuth and pitch angles from the camera to the target are calculated using a mathematical function relationship based on the azimuth and pitch angles of the camera.

5. The fire monitoring method based on a monocular camera and a digital surface model according to claim 1, characterized in that: The step 4 obtains the latitude and longitude coordinates and elevation information of all pixel points of the camera in azimuth, calculates the distance and elevation difference between each pixel point and the camera position, infers the pitch angle from the camera to each pixel point, and compares it with the pitch angle from the camera to the target, thereby obtaining the actual coordinates of the target.

6. The fire monitoring method based on a monocular camera and a digital surface model according to claim 2, characterized in that: The camera calibration and camera correction are specifically as follows: 1) Camera calibration: prepare a 2D checkerboard calibration plate and use the camera to shoot the calibration object from multiple angles. Use Zhang Zhengyou's calibration method to calibrate the camera and solve the camera's intrinsic matrix, extrinsic matrix and distortion parameters. The intrinsic matrix includes the pixel focal length. , and the pixel coordinates of the principal point ; 2) Camera calibration: Apply the distortion correction algorithm to correct the image based on the parameters obtained from camera calibration.

7. The fire monitoring method based on a monocular camera and a digital surface model according to claim 3, characterized in that: The fire point identification is as follows: 1) Target detection model training and optimization: prepare target detection hotspot data set and perform data cleaning, convert the data set format into YOLO format, divide the training set and validation set according to a certain ratio; use YOLOv5 target detection network for model training, adjust the model training hyperparameters in time, and optimize the training results; 2) Model testing and deployment. After completing the target detection model training, it is tested on an independent test set to evaluate the model performance and deployed for practical applications to realize monocular camera fire point recognition and obtain the predicted box position of the fire point in the image.

8. A fire monitoring method based on a monocular camera and a digital surface model according to claim 4, characterized in that: The calculation of the target azimuth and elevation angle is as follows: 1) Obtain the coordinates of the icon image, input the image captured by the monocular camera into the pre-trained YOLOv5 model, perform inference analysis on the image, identify the fire points in the image and output the prediction box of each fire point; then calculate the pixel coordinates of the center point of the prediction box, and further convert the pixel coordinates of the prediction box into image coordinates based on the image resolution. ; 2) Calculate the azimuth and pitch angles from the camera's principal point to the target, and obtain the camera azimuth angle provided by the monocular camera gimbal , Pitch angle And the world coordinates of the camera; based on the target image coordinates and pixel focal length , , calculate the azimuth from the camera to the target and pitch angle : 。 9. The fire monitoring method based on a monocular camera and a digital surface model according to claim 5, characterized in that: The actual coordinates of the calculated target are as follows: 1) Calculate the pitch angle of all pixels in the direction of the camera position and target azimuth on the digital surface model. In three-dimensional space, the longitude, latitude and elevation of the camera are , , , taking the camera as the origin, obtain the camera's azimuth on the digital elevation model image α 0 and the elevation of the digital surface model, record ; For all pixels, calculate the camera coordinates and pixel coordinates The distance in the horizontal direction and the vertical elevation difference ; 2) Calculate the actual coordinates of the target, and then find the angles between each pixel point in the three-dimensional space and each pixel point in the vertical direction of the camera in the direction of the camera azimuth angle α. , if the angle calculated by the two pixels is at the pitch angle from the camera to the target On both sides, the average of the longitude and latitude of the two corresponding pixels is the actual coordinates of the target.

10. A fire monitoring method based on a monocular camera and a digital surface model according to claim 9, characterized in that: Calculate camera coordinates and pixel coordinates The distance in the horizontal direction : ； ； ； in, , are the longitude and latitude of the camera, 、 are the longitude and latitude of the pixel coordinates respectively; is the distance between two points, is the mean radius of the Earth; Calculate camera coordinates and pixel coordinates The difference in elevation in the vertical direction : ； in, is the elevation of the pixel, is the camera elevation. Calculate camera coordinates and pixel coordinates Pitch angle : 。

Images