An auxiliary decision-making method and system for forest fire fighting operations based on an airborne optoelectronic pod

Abstract

The present invention discloses a method and system for assisting decision-making in forest fire fighting operations based on an airborne optoelectronic pod, which relates to the technical field of forest fire prevention and control. The technical key points of the present invention include: collecting continuous frame images, detecting whether there is a fire in the current frame; identifying the fire front for the image frames where a fire is detected; calculating and obtaining the key points of the fire front contour curve based on the morphological characteristics of the fire front; calculating the positions of the key points relative to the center of the image plane of the frame; determining the control amount of the servo system required for the laser rangefinder to align with the key points according to the above positions; ranging the key points based on the control amount of the servo system; calculating and obtaining the position of the fire front based on the relative positions obtained by ranging; predicting the spread of the fire according to the position of the fire front and other information; and performing real-time planning and dynamic update of the retreat route of the fire fighting force according to the position of the fire front and the prediction result of the spread. The present invention can help commanders quickly and accurately grasp the dynamics of the fire scene, thereby improving the efficiency of forest fire emergency response.

Description

Technical Field

[0001] The present invention relates to the technical field of forest fire prevention and control, and particularly to a method and system for assisting decision-making in forest fire fighting operations based on an airborne optoelectronic pod. Background Art

[0002] Traditional forest fire monitoring methods mainly rely on ground stations and satellite remote sensing. These methods have deficiencies in spatial resolution and real-time performance, and it is difficult to provide refined fire situation information, resulting in low efficiency of fire fighting operations and even missing the best opportunity for fire suppression. Compared with ground stations and satellite remote sensing, airborne platforms have high flexibility, can quickly cover a large-scale fire area, and can adjust the observation angle and range according to requirements.

[0003] An airborne optoelectronic pod equipped with an optical sensor, an infrared sensor, a laser rangefinder, a servo control system, and an integrated inertial navigation system is very suitable for forest fire monitoring. For example, abnormal hot spots in the monitoring area are combined with laser ranging information and the pose information of the optoelectronic pod to achieve rough positioning of the fire source. However, the current fire monitoring methods based on airborne monitoring pods have a low degree of automation, often reducing the three-dimensional surface characteristics of fire spread to one-dimensional point characteristics, and no efficient system for assisting decision-making in forest fire fighting operations has been formed. Summary of the Invention

[0004] In view of the above problems, the present invention proposes a method and system for assisting decision-making in forest fire fighting operations based on an airborne optoelectronic pod.

[0005] According to one aspect of the present invention, a method for assisting decision-making in forest fire fighting operations based on an airborne optoelectronic pod is proposed. The method includes the following steps:

[0006] Collect continuous frame images and perform a first processing on the frame images to detect whether there is a fire in the current frame;

[0007] Perform a second processing on the image frames detected with a fire to identify the fire front; wherein, the fire front is defined as the part where the flame extends forward, and its contour curve is the boundary line between the burning area and the unburned area;

[0008] Perform a third processing based on the morphological characteristics of the fire front to obtain key points on the contour curve of the fire front;

[0009] Calculate the position of the key points relative to the center of the image plane of the frame image to obtain relative pixel coordinates;

[0010] Perform a fourth processing on the relative pixel coordinates to determine the control amount of the servo system required for the laser rangefinder on the airborne optoelectronic pod to align with each key point, and perform ranging on each key point based on the control amount of the servo system;

[0011] Perform the fifth processing based on the relative positions of each key point obtained by ranging from the laser rangefinder, and calculate the position of the fire front;

[0012] Predict the spread of the fire according to the position of the fire front and other information;

[0013] Plan and dynamically update the evacuation route of the fire fighting force in real time according to the position of the fire front and the prediction result of the spread;

[0014] Further, the first processing of the frame image to detect whether there is a fire in the current frame includes: the frame image includes a visible light image and an infrared image, extract features of different scales from the visible light image and the infrared image respectively; perform weighted fusion on the two features to obtain a fused feature; splice the fused features of all scales to obtain a fused feature map of the fused multimodal image; input the fused feature map into a fire detection model based on deep learning for fire detection, and detect and determine whether there is a fire in the current frame.

[0015] Further, the second processing of the image frame detected with fire includes: perform image segmentation on the image frame detected with fire to obtain a binary image; use the Canny algorithm to extract the edge of the burning area in the binary image; calculate the optical flow between adjacent frames, and then obtain the optical flow velocity; identify the curve formed by the pixels with the optical flow velocity greater than the preset velocity threshold as the fire front contour curve.

[0016] Further, the third processing based on the morphological features of the fire front to obtain the key points of the fire front contour curve includes: calculate the curvature value of each point along the fire front curve contour, sort all the curvature values, and assign an index i to each point; among them, the point with the largest curvature value is assigned an index value of 1, and the point with the smallest curvature value is assigned an index value of n, where n represents the number of fire points on the fire front curve contour; assume that the number of continuously detectable points of the laser rangefinder on the airborne optoelectronic pod is P, and the points on the fire front contour curve with the index value less than or equal to P are the key points of the fire front contour curve to be measured by the laser rangefinder.

[0017] Further, the fourth processing of the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align with each key point includes: convert the relative pixel coordinates into normalized camera coordinates; calculate the angular deviation in the horizontal and vertical directions in the normalized camera coordinate system; according to the angular deviation, calculate the servo system control amount according to the following formula:

[0018] Δα x =k x ·θ x ,Δα y =k y ·θ y

[0019] where Δα x represents the adjustment angle of the servo system in the horizontal direction, and Δα y represents the adjustment angle of the servo system in the vertical direction, and k x and k y are the gain coefficients of the servo system; θ x , θ y respectively represent the angle deviations in the horizontal and vertical directions.

[0020] Furthermore, the relative positions of each key point obtained based on the ranging with respect to the laser rangefinder are subjected to a fifth processing, and calculating the position of the fire front includes:

[0021] If the number of continuously detectable points P of the laser rangefinder is equal to 1, calculate the pixel sampling distance through the distance from the laser rangefinder to the ground, the pixel size, and the focal length. Taking the key point of the fire front as the reference point, calculate the actual relative positions of each pixel with respect to the reference point through the pixel distances and the pixel sampling distance between the remaining pixels and the key point; calculate the actual positions of the remaining points on the fire front curve contour in combination with the actual coordinates of the key point;

[0022] If the number of continuously detectable points P of the laser rangefinder is equal to 2, calculate the conversion ratio factor between the pixel distance and the actual distance based on the pixel distances and the actual distances of the two key points; solve the equation of the straight line where the two key point pixels are located; calculate the pixel distances from the remaining points on the curve contour to the straight line; calculate the actual distances from the remaining points on the fire front curve contour to the straight line according to the conversion ratio factor between the pixel distance and the actual distance; calculate the actual positions of the remaining points on the fire front curve contour according to the actual positions of the points on the straight line and the actual distances from the remaining points on the fire front curve contour to the straight line;

[0023] If the number of continuously detectable points P of the laser rangefinder is greater than or equal to 3, then for each point on the fire front curve contour, successively select the three key points closest to it to establish a local plane; through the perspective projection relationship between the pixel plane and the local plane, map the two-dimensional pixel coordinates of the remaining points on the fire front curve contour to the three-dimensional space, thereby determining the actual positions of the remaining points on the fire front curve contour.

[0024] Further, the real-time planning and dynamic update of the fire-fighting force retreat route according to the position of the fire front and the spread prediction result include: calculating the evacuation capacity index (ECI) of each grid in the grid map of the area based on the terrain characteristics of the area around the fire scene; calculating the comprehensive safety index (CSI) of each grid based on the current fire front position and the spread prediction result; establishing a potential safety path evaluation function by combining the evacuation capacity index (ECI) and the comprehensive safety index (CSI); taking the maximization of the potential safety path evaluation function as the objective function and using the A* algorithm to solve for the retreat route; dynamically updating the evacuation capacity index (ECI) and the comprehensive safety index (CSI) of each grid in combination with the real-time fire front position and the fire spread situation of the optoelectronic pod, and adjusting the retreat route in real time.

[0025] Further, the calculation formula of the evacuation capacity index (ECI) is:

[0026]

[0027] where S(i,j) represents the slope of grid (i,j); D(i,j) represents the obstacle density in the grid; α 1 and α 2 represent adjustment coefficients;

[0028] The calculation formula of the comprehensive safety index (CSI) is:

[0029] CSI(i,j) = ω 1 ·TSI(i,j) + ω 2 ·SSI(i,j)

[0030] where TSI represents the time safety index, T safe (i,j) represents the safety time of grid (i,j) before the fire spreads; T arrival represents the time required for the fire-fighting force to reach the grid; SSI represents the space safety index, d fire (i,j) represents the distance from grid (i,j) to the fire front; ω 1 and ω 2 are weights.

[0031] Further, the expression of the potential safety path evaluation function is:

[0032] PathScore(i,j) = λ 1 ·ECI(i,j) + λ 2 ·CSI(i,j)

[0033] where λ 1 and λ 2 represent the weights of the evacuation capacity index and the comprehensive safety index.

[0034] Further, the other information includes: local meteorological information of the fire scene, terrain and landform information, and combustible property information.

[0035] According to another aspect of the present invention, a forest fire fighting operation auxiliary decision-making system based on an airborne optoelectronic pod is proposed. The system includes:

[0036] A fire situation detection module configured to collect consecutive frame images and perform a first processing on the frame images to detect whether there is a fire in the current frame;

[0037] A fire front recognition module configured to perform a second processing on the image frames where a fire is detected to identify the fire front; wherein, the fire front is defined as the part where the flame spreads forward, and its contour curve is the boundary line between the burning area and the unburned area;

[0038] A fire front positioning module configured to perform a third processing based on the morphological characteristics of the fire front to obtain key points of the fire front contour curve; calculate the positions of the key points relative to the image plane center of the frame image to obtain relative pixel coordinates; perform a fourth processing on the relative pixel coordinates to determine the servo system control amounts required for the laser rangefinder on the airborne optoelectronic pod to align with each key point, and perform ranging on each key point based on the servo system control amounts; perform a fifth processing based on the relative positions of each key point obtained by ranging and the laser rangefinder to calculate and obtain the position of the fire front;

[0039] A spread prediction module configured to predict the spread of the fire according to the position of the fire front and other information;

[0040] A safe path planning module configured to perform real-time planning and dynamic update of the retreat route of the fire fighting force according to the position of the fire front and the spread prediction result.

[0041] The beneficial technical effects of the present invention are:

[0042] The present invention provides a method and system for assisting decision-making in forest fire fighting operations based on an airborne optoelectronic pod. The aim is to help commanders quickly and accurately grasp the dynamics of the fire scene, formulate effective fire-fighting strategies, reduce losses caused by the fire, and improve the efficiency and effectiveness of forest fire emergency response through efficient data collection, processing, and fusion. First, continuous frame images are collected and the frame images are processed for the first time to detect whether there is a fire in the current frame. By fusing multi-modal image data of the fire scene for fire detection, the false alarm rate and missed alarm rate of fire detection are reduced. Then, the image frames with detected fire are processed for the second time to identify the fire front. Based on the morphological characteristics of the fire front, a third processing is carried out to obtain the key points of the fire front contour curve, calculate the positions of the key points relative to the center of the image plane of the frame image, and obtain the relative pixel coordinates. The relative pixel coordinates are processed for the fourth time to determine the servo system control amounts required for the laser rangefinder on the airborne optoelectronic pod to align with each key point, and range measurement is carried out on each key point based on the servo system control amounts. Based on the relative positions of each key point obtained by range measurement and the laser rangefinder, a fifth processing is carried out to calculate the position of the fire front. The above perspective projection method based on the adaptive selection of fire front key points can quickly obtain the three-dimensional coordinates of the fire front in one imaging, realize the global positioning of the fire front, and break through the limitation of single-point positioning of the laser. Then, the spread of the fire is predicted according to the position of the fire front and other information. The retreat route of the fire-fighting force is planned and dynamically updated in real time according to the position of the fire front and the spread prediction result. The above provides an optimized path planning scheme for the operation and retreat of the fire-fighting force by fusing multi-source data and intelligent algorithms, ensuring the safety of fire-fighting operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention can be better understood by referring to the following description in conjunction with the accompanying drawings, which are included in this specification together with the following detailed description and form a part of this specification, and are used to further illustrate the preferred embodiments of the present invention and explain the principles and advantages of the present invention.

[0044] Figure 1 It is a flowchart of a method for assisting decision-making in forest fire fighting operations based on an airborne optoelectronic pod according to an embodiment of the present invention;

[0045] Figure 2 It is another flowchart of a method for assisting decision-making in forest fire fighting operations based on an airborne optoelectronic pod according to an embodiment of the present invention;

[0046] Figure 3 It is an example diagram of the processing results of image segmentation, edge detection, and fire front recognition of an image in an embodiment of the present invention.

[0047] Figure 4Schematic diagram of local fire front positioning through the projection relationship between the pixel plane and the local plane in the embodiment of the present invention.

[0048] Figure 5 Flow chart of the spread prediction model that can realize real-time calibration of input parameters in the embodiment of the present invention.

[0049] Figure 6 Example diagram of fire spread situation and evacuation path planning in the embodiment of the present invention. Detailed implementation manners

[0050] In order to enable those skilled in the art to better understand the solution of the present invention, the exemplary embodiments or examples of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments or examples are only part of the embodiments or examples of the present invention, rather than all of them. Based on the embodiments or examples in the present invention, all other embodiments or examples obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0051] The embodiment of the present invention provides a forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod, as Figures 1 - 2 shown, including the following steps:

[0052] S1. Collect continuous frame images and perform the first processing on the frame images to detect whether there is a fire in the current frame; the specific steps include: the frame images include visible light images and infrared images, extract features of different scales from the visible light images and infrared images respectively; perform weighted fusion on the two features to obtain fused features; splice the fused features of all scales to obtain a fused feature map of the fused multi-modal image; input the fused feature map into a fire detection model based on deep learning for fire detection to detect and determine whether there is a fire in the current frame;

[0053] S2. Perform the second processing on the image frames detected with a fire to identify the fire front; wherein, the fire front is defined as the part where the flame spreads forward, and its contour curve is the boundary line between the burning area and the unburned area; the specific steps include: perform image segmentation on the image frames detected with a fire to obtain a binary image; use the Canny algorithm to extract the edge of the burning area in the binary image; calculate the optical flow between adjacent frames, and then obtain the optical flow velocity; identify the curve formed by the pixels with the optical flow velocity greater than the preset velocity threshold as the fire front contour curve;

[0054] S3. Perform the third processing based on the morphological characteristics of the fire front to obtain the key points of the fire front contour curve. The specific steps include: calculating the curvature value of each point along the fire front curve contour, sorting all the curvature values, and assigning an index i to each point. Among them, the point with the largest curvature value is assigned an index value of 1, and the point with the smallest curvature value is assigned an index value of n, where n represents the number of fire points on the fire front curve contour. Assume that the number of continuously detectable points of the laser rangefinder on the airborne optoelectronic pod is P. The key points of the fire front contour curve to be measured by the laser rangefinder are the points on the fire front curve contour whose index values are less than or equal to P.

[0055] S4. Calculate the position of the key points relative to the image plane center of this frame of image to obtain the relative pixel coordinates. Perform the fourth processing on the relative pixel coordinates to determine the servo system control amounts required for the laser rangefinder on the airborne optoelectronic pod to align with each key point, and perform ranging on each key point based on the servo system control amounts. The specific steps include: converting the relative pixel coordinates into normalized camera coordinates; calculating the angular deviations in the horizontal and vertical directions in the normalized camera coordinate system; calculating the servo system control amounts based on the angular deviations.

[0056] S5. Perform the fifth processing based on the relative positions of each key point obtained by ranging and the laser rangefinder to calculate the position of the fire front. The specific steps include: if the number of continuously detectable points P of the laser rangefinder is equal to 1, calculate the pixel sampling distance through the distance from the laser rangefinder to the ground, the pixel size, and the focal length. Taking the fire front key point as the reference point, calculate the actual relative positions of the remaining pixels and the reference point through the pixel distances between the remaining pixels and the key point and the pixel sampling distance; calculate the actual positions of the remaining points on the fire front curve contour in combination with the actual coordinates of the key points. If the number of continuously detectable points P of the laser rangefinder is equal to 2, calculate the pixel-to-actual distance conversion ratio factor based on the pixel distance and the actual distance between the two key points; solve the equation of the straight line where the two key point pixels are located; calculate the pixel distances from the remaining points on the curve contour to the straight line; calculate the actual distances from the remaining points on the fire front curve contour to the straight line according to the pixel-to-actual distance conversion ratio factor; calculate the actual positions of the remaining points on the fire front curve contour based on the actual positions of the points on the straight line and the actual distances from the remaining points on the fire front curve contour to the straight line. If the number of continuously detectable points P of the laser rangefinder is greater than or equal to 3, for each point on the fire front curve contour, successively select the three key points closest to it to establish a local plane; through the perspective projection relationship between the pixel plane and the local plane, map the two-dimensional pixel coordinates of the remaining points on the fire front curve contour to three-dimensional space, so as to determine the actual positions of the remaining points on the fire front curve contour.

[0057] S6. Predict the spread of the fire according to the position of the fire front and other information.

[0058] S7. Plan and dynamically update the evacuation route of the fire fighting force in real time according to the position of the fire front and the spread prediction result. The specific steps include: calculating the evacuation ability index of each grid in the grid map of the area based on the terrain characteristics of the surrounding area of the fire scene; calculating the comprehensive safety index of each grid based on the current fire front position and the spread prediction result; establishing a potential safety path evaluation function by combining the evacuation ability index and the comprehensive safety index; taking the maximization of the potential safety path evaluation function as the objective function, and using the A* algorithm to solve and obtain the evacuation route; combining the real-time fire front position and the fire spread situation of the optoelectronic pod, dynamically updating the evacuation ability index and the comprehensive safety index of each grid, and adjusting the evacuation route in real time.

[0059] The method starts from S1. In S1, continuous frame images are collected and the frame images are processed for the first time to detect whether there is a fire in the current frame.

[0060] According to the embodiments of the present invention, first, taking the visible light and infrared images obtained by the optoelectronic pod as inputs, fusing the features of multi-modal images, and detecting whether there is a fire in the current field of view in real time based on the fused images. It should be noted that the sensors carried by the airborne optoelectronic pod should include a visible light camera, an infrared camera, a laser rangefinder, an inertial navigation system, etc.

[0061] As an example, the specific method for fusing visible light and infrared images is as follows:

[0062] S11. Use the convolutional neural network CNN to process the infrared image and the visible light image I respectively RGB , using n convolutional layers L with different sizes of convolutional kernels 1 ,L 2 ,……L n ; The output feature map of each convolutional layer is: Among them, and are the feature maps of the infrared and visible light images in the i-th convolutional layer respectively;

[0063] S12. Use bilinear interpolation to adjust the feature maps to the same size H×W;

[0064] S13. For each scale i, fuse the infrared feature map and the visible light feature map in a weighted summation manner:

[0065]

[0066] S14. Stitch the fused feature maps of all scales to obtain the final fused feature map:

[0067]

[0068] As an example, the fire detection model based on deep learning uses a lightweight deep learning model, which takes MobileNetV3 as the backbone network. The specific structure is as follows: The input data first passes through a standard convolutional layer; then it goes through n depthwise separable convolutional blocks, each block including a depthwise convolution and a pointwise convolution, followed by Batch Normalization and Hard Swish; global average pooling is used to reduce the spatial dimension of the feature map to 1 to obtain a fixed-length feature vector; it goes through m fully connected layers to further process the extracted features; the output layer uses the Sigmoid activation function to output the fire detection result in a binary classification manner.

[0069] Then, S2 is executed. In S2, the image frame with detected fire is processed for the second time to identify the fire front.

[0070] According to the embodiment of the present invention, for the image frame with detected fire, the burning area is segmented, its contour is extracted, and the fire front and the fire rear front are identified; among them, the fire front is the part where the flame spreads forward and the boundary line between the burning area and the unburned area, and the fire rear front is the boundary line between the burning area and the burned area.

[0071] As an example, the specific steps of burning area segmentation, contour extraction, and fire front identification include:

[0072] S21. Perform image segmentation with a custom threshold T:

[0073]

[0074] Among them, p(i) is the pixel gray probability, u 1 and u 2 are the average values before and after the segmentation point, i is the gray level, and L is the maximum value of the gray level; a binary image is output, where the white area represents the detected burning area, and the black area represents the burned area and the unburned area;

[0075] S22. Use the Canny algorithm to extract the edge of the burning area;

[0076] S23. Use the Lucas-Kanade method to calculate the optical flow between adjacent frames I(t) and I(t + 1). The optical flow equation is:

[0077] I x u + I y v + I t = 0

[0078] Among them, I(x) and I(y) are the spatial gradients of the image respectively, and I(t) is the temporal gradient. By solving the optical flow equation, the optical flow components (u, v) of each pixel are obtained;

[0079] S24. Calculate the optical flow velocity:

[0080]

[0081] S25. The pixels with an optical flow velocity greater than the velocity threshold V th constitute a curve which is the fire front. Figure 3 shows the fire front recognition process and results.

[0082] Then, execute S3. In S3, based on the morphological features of the fire front, perform a third processing to obtain the key points of the fire front contour curve.

[0083] According to the embodiment of the present invention, calculate the curvature of each point along the fire front contour, sort them according to the calculated curvature values, and assign an index i to each point. The point with the largest curvature value is assigned an index value of 1, and the point with the smallest curvature value is assigned an index value of n, which is the number of fire points on the fire front contour. Assume that the number of points that can be continuously detected by the laser rangefinder of the optoelectronic pod is P. The points on the fire front contour with an index value less than or equal to P are the key points of the fire front to be detected by the laser rangefinder. It should be noted that the number of key points of the fire front to be detected by the laser rangefinder is determined by the hardware attributes of the laser rangefinder.

[0084] As an example, the method for calculating the curvature of each pixel point on the fire front contour is as follows:

[0085]

[0086] where x′ and y′ are the reciprocals of the current point, and x″ and y″ are the second derivatives of the current point; the greater the curvature, the higher the degree of bending at the position of this point.

[0087] Then, execute S4. In S4, calculate the position of the key point relative to the center of the image plane of this frame, and obtain the relative pixel coordinates; perform a fourth processing on the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align with each key point, and perform ranging on each key point based on the servo system control amount

[0088] According to the embodiment of the present invention, after extracting the key points based on the morphological features of the fire front, calculate the position of the key point with index P relative to the center of the image plane, determine the servo system control amount required for the laser rangefinder to align with this point, control the optoelectronic pod servo system to achieve automatic alignment and ranging, and sequentially detect key points with index values of P - 1, P - 2, etc., until all ranging is completed. Ensure that the high-priority key points with larger curvatures can be detected with the minimum time delay and quickly proceed to the next operation.

[0089] As an example, the specific steps for calculating the position of the key point with index P relative to the center of the image plane and determining the servo system control amount required for the laser rangefinder to align with this point include:

[0090] S41. Calculate the position of the key point with index P relative to the image plane center, and obtain the relative pixel coordinates.

[0091] S42. Convert the relative pixel coordinates (u, v) to the normalized camera coordinates (x n , y n ):

[0092]

[0093] where f x is the focal length of the camera in the horizontal direction; f y is the focal length of the camera in the vertical direction.

[0094] S43. In the normalized camera coordinate system, calculate the angular deviations θ x , θ y :

[0095] θ x = arctan(x n ), θ y = arctan(y n ).

[0096] S44. Convert the calculated angular deviations θ x and θ y into the control quantities of the servo system:

[0097] Δα x = k x ·θ x , Δα y = k y ·θ y

[0098] where Δα x is the adjustment angle of the servo system in the horizontal direction, Δα y is the adjustment angle of the servo system in the vertical direction, and k x and k y are the gain coefficients of the servo system.

[0099] Then execute S5. In S5, based on the relative positions of the key points obtained by ranging with the laser rangefinder, perform the fifth processing to calculate and obtain the position of the fire front.

[0100] According to the embodiments of the present invention, according to the number of continuously detectable points P of the laser rangefinder, the specific scheme for determining the fire front positioning includes:

[0101] S51. If P = 1, calculate the pixel sampling distance based on the distance from the camera to the ground, the pixel size, and the focal length. Using the key point of the fire front with an index value of 1 as the reference point, calculate the relative position of each pixel to the reference point based on the pixel distance between the remaining pixels and the key point and the pixel sampling distance. Combine the absolute coordinates measured by the inertial navigation system, the extrinsic parameters between the laser and the inertial navigation system, the relative position between the key point and the laser, and the relative position between the remaining pixel points on the fire front and the key point to achieve the positioning of the fire front.

[0102] As an example, the specific steps of the fire front positioning corresponding to P = 1 in S51 include:

[0103] S511. Based on the distance H from the camera to the ground and the focal length f of the camera in the horizontal direction x and the focal length f in the vertical direction y , calculate the pixel sampling distance:

[0104]

[0105] where d x is the actual distance corresponding to each pixel in the horizontal direction, d y is the actual distance corresponding to each pixel in the vertical direction, s x and s y are the pixel sizes in the horizontal and vertical directions respectively;

[0106] S512. Calculate the actual relative position of other pixel points (x i , y i ) on the fire front to the key point (X 1 , Y 1 , Z 1 ):

[0107] ΔX i =(x i -x 1 )·d x , ΔY i =(y i -y 1 )·d y

[0108] S513. Combine the actual coordinates of the key point to calculate the actual ground coordinates of the remaining points:

[0109] X i =X 1 +ΔX i , Y i =Y 1 +ΔY i

[0110] S514. Repeat S512 - S513 to calculate for each pixel on the fire front, and obtain the actual positions of the remaining points.

[0111] S52. If P = 2, obtain the conversion relationship between the two key points through their actual distance and pixel distance; taking any key point as a reference point, combine the conversion relationship between pixel distance and actual distance to obtain the relative positions of each pixel point on the straight line where the two key points are located with respect to the key points, and finally obtain the relative positions of each pixel point on the fire front with respect to the key points according to the distances between the remaining pixels on the fire front and the straight line where the two key points are located; combine the absolute coordinates of itself measured by the inertial navigation system, the external parameters of the laser and the inertial navigation system, the relative positions of the key points and the laser, and the relative positions of the remaining pixel points on the fire front with respect to the key points to achieve fire front positioning.

[0112] As an example, the specific steps of fire front positioning corresponding to P = 2 in S52 include:

[0113] S521. Calculate the pixel - to - actual distance conversion scale factor α based on the pixel distance and actual distance of the two key points:

[0114]

[0115] where (x 1 , y 1 ), (x 2 , y 2 ) are the pixel coordinates of the two key points respectively, and (X 1 , Y 1 ), (X 2 , Y 2 ) are the actual ground coordinates of the key points actually measured through pose transformation respectively;

[0116] S522. Solve the equation of the straight line where the two key point pixels are located:

[0117] Ax + By + V = 0

[0118] where A, B, C are parameters to be solved;

[0119] S523. Calculate the pixel distance d from other pixels (x i , y i ) on the fire front to the straight line:

[0120]

[0121] S524. Calculate the actual distance D from any pixel on the fire front to this straight line according to the pixel - to - actual distance conversion scale factor α:

[0122] D = α·d

[0123] S525. Calculate the actual ground coordinates of each point on the fire front based on the actual ground coordinates of each point on the straight line and the actual distances from each point on the fire front to the straight line.

[0124] S53. If P >= 3, for each pixel point on the fire front, successively select three most adjacent key points to establish a local plane. Then, through the perspective projection relationship between the pixel plane and this local plane, map the two-dimensional pixel coordinates of the remaining points on the fire front to three-dimensional space, and finally achieve fire front positioning.

[0125] As an example, the specific steps of the fire front positioning corresponding to P >= 3 in S53 include:

[0126] S531. Select three key points (X 1 , Y 1 , Z 1 ), (X 2 , Y 2 , Z 2 ), (X 3 , Y 3 , Z 3 ) on the fire front to define the local plane, and calculate the plane normal vector n through the vector cross product:

[0127]

[0128] S532. Using the normal vector n = (a, b, c) and a point (X 1 , Y 1 , Z 1 ) on the plane, the plane equation can be expressed as:

[0129] ax + by + cz + d = 0

[0130] where d is the constant term, and is obtained by substituting a point on the plane:

[0131] d = -(aX 1 + bY 1 + cZ 1 )

[0132] S533. Convert the world coordinates to camera coordinates through the camera internal parameter matrix K and the external parameter matrix (including the rotation matrix R and the translation vector T);

[0133] S534. For the pixel coordinates (u, v) of non-key points, map them back to three-dimensional space from the two-dimensional pixel plane by solving the inverse transformation of the projection matrix:

[0134]

[0135] where λ is the scale factor, and (X, Y, Z) is solved through the inverse projection relationship;

[0136] S535. The solved three-dimensional coordinates (X, Y, Z) may not be exactly located on the local plane and need to be projected back onto the local plane, that is, find the nearest point (X p , Y p , Z p ) that satisfies:

[0137] ax + by + cz + d = 0

[0138] (X p , Y p , Z p ) is the final three-dimensional coordinate of the fire front pixel point.

[0139] Figure 4 Shows the local fire front positioning map through the projection relationship between the pixel plane and the local plane.

[0140] Then execute S6. In S6, predict the fire spread based on the position of the fire front and other information.

[0141] According to the embodiments of the present invention, the spread prediction can be based on the fire front position, local meteorological information of the fire scene, topographic and geomorphic information, and combustible property information to obtain the fire line position after δt. The spread prediction method can update the spread prediction result through real-time data and calibrate the input parameters of the spread prediction by minimizing the real-time observation error of the fire front, including but not limited to the water content of combustibles and high-resolution wind fields. Among them, the local meteorological information of the fire scene can be obtained by crawling relevant authoritative websites, or by real-time collection using airborne multi-element meteorological measurement devices, or by real-time transmission with temporary meteorological stations around the fire scene. There are differences in the spatio-temporal resolution of meteorological data obtained by different means.

[0142] As an example, the steps of predicting the fire spread based on the position of the fire front and other information include:

[0143] S61. Construct a spread prediction model that combines the Rothermel spread speed model and the level set method (as a two-dimensional extended model);

[0144] S62. At time t0, input the fire front position observed at time t0, local real-time meteorological data of the fire scene, the collected slope, aspect, altitude, combustible type, and water content of combustibles of the fire scene, and predict the fire front position after time t1;

[0145] S63. At time t1, first calculate the Hausdorff distance between the currently observed fire front position and the predicted result at time t0 as the prediction error, then minimize the error through the differential evolution algorithm to achieve the calibration of the input parameters, and finally predict the fire front position at time t2 based on the calibrated parameters; predict the fire front positions at subsequent times according to the above process. The flow chart of the spread prediction method is as Figure 5 shown.

[0146] Finally, execute S7. In S7, based on the position of the fire front and the spread prediction result, plan and dynamically update the evacuation route of the fire fighting force in real time.

[0147] According to the embodiment of the present invention, starting from the current position of the fire fighting force, based on the real-time fire front position information obtained by the optoelectronic pod and the spread situation deduction information obtained in step six, plan and dynamically update the evacuation route of the fire fighting force in real time. The specific steps include:

[0148] S71. Calculate the evacuation capacity index ECI of each grid according to the terrain features (including slope, obstacles, road conditions), and measure the passability of the grid through ECI:

[0149]

[0150] Among them, S(i, j) represents the slope of the grid (i, j); D(i, j) represents the obstacle density in the grid; α 1 and α 2 represent adjustment coefficients;

[0151] S72. Based on the current fire front position and the spread prediction result, calculate the comprehensive safety index (CSI) of each grid in the area. This comprehensive safety index includes two parts: the time safety index (TSI) and the space safety index (SSI):

[0152] CSI(i, j) = ω 1 ·TSI(i, j) + ω 2 ·SSI(i, j)

[0153] Among them, T safe (i, j) represents the safety time of the grid (i, j) before the arrival of the fire spread, and T arrival represents the time required for the fire fighting force to reach the grid; d fire (i, j) represents the distance from the grid (i, j) to the fire front, and ω 1 and ω 2 are weights.

[0154] S73. Combine the evacuation capacity index and the comprehensive safety index to establish a potential safety path evaluation function for the fire fighting force safe evacuation path planning algorithm:

[0155] PathScore(i,j)=λ 1 ECI(i,j)+λ 2 CSI(i,j)

[0156] Among them, λ 1 and λ 2 is the weight of evacuation capacity and safety index;

[0157] S74. Use the A* algorithm to balance the safety of the path and the evacuation efficiency, maximize the evaluation function PathScore(i,j), and generate the optimal evacuation path for the firefighting force:

[0158]

[0159] S75. Combined with the real-time fire front position and fire spread situation of the optoelectronic pod, dynamically update the evacuation capacity index and comprehensive safety index of each grid, and adjust the retreat path in real time. The updated retreat path meets the new safety requirements while maintaining the continuity and effectiveness of the path as much as possible. Figure 6 shown.

[0160] Another embodiment of the present invention provides a forest fire fighting operation auxiliary decision system based on an airborne optoelectronic pod, the system comprising:

[0161] A fire detection module is configured to collect continuous frame images and perform a first processing on the frame images to detect whether there is a fire in the current frame;

[0162] A fire front identification module is configured to perform a second processing on the image frame where the fire is detected to identify the fire front; wherein the fire front is defined as the portion where the flame extends forward, and its contour curve is the boundary between the burning area and the unburned area;

[0163] The fire front positioning module is configured to perform a third processing based on the morphological features of the fire front to obtain key points of the fire front contour curve; calculate the position of the key points relative to the center of the image plane of the frame image to obtain relative pixel coordinates; perform a fourth processing on the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align each key point, and perform distance measurement on each key point based on the servo system control amount; perform a fifth processing based on the relative position of each key point obtained by the distance measurement and the laser rangefinder to calculate the position of the fire front;

[0164] A spread prediction module configured to predict the spread of a fire based on the position of the fire front and other information;

[0165] A safe path planning module configured to perform real-time planning and dynamic update of the retreat route of fire-fighting forces based on the position of the fire front and the spread prediction result.

[0166] The functions of the forest fire fighting operation auxiliary decision-making system based on an airborne optoelectronic pod in this embodiment can be illustrated by the foregoing forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod. Therefore, for the parts not described in detail in this embodiment, reference may be made to the above method embodiments, which will not be elaborated herein.

[0167] Although the present invention has been described based on a limited number of embodiments, those skilled in the art in this technical field will understand that other embodiments can be conceived within the scope of the present invention thus described. For the scope of the present invention, the disclosure made for the present invention is illustrative rather than restrictive, and the scope of the present invention is defined by the appended claims.

Claims

1. A forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod, characterized in that: The following steps are involved: Collect continuous frame images and perform the first processing on the frame images to detect whether there is a fire in the current frame; Processing the image frame where the fire is detected for the second time to identify the fire front; wherein the fire front is defined as the portion of the flame extending forward, and its contour curve is the boundary between the burning area and the unburned area; The third processing is performed based on the morphological characteristics of the fire front to obtain the key points of the fire front contour curve; Calculate the position of the key point relative to the center of the image plane of the frame image to obtain the relative pixel coordinates; Processing the relative pixel coordinates for a fourth time to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align the laser rangefinder with each key point, and measuring the distance of each key point based on the servo system control amount; The fifth processing is performed based on the relative position of each key point obtained by ranging and the laser rangefinder to calculate the position of the fire front; including: if the number of continuously detectable points P of the laser rangefinder is equal to 1, the pixel sampling distance is calculated by the distance from the laser rangefinder to the ground, the pixel size and the focal length, and the key point of the fire front is used as a reference point, and the actual relative position of each pixel and the reference point is calculated by the pixel distance between the remaining pixels and the key point and the pixel sampling distance; the actual position of the remaining points on the curve contour of the fire front is calculated in combination with the actual coordinates of the key point; If the number of continuously detectable points P of the laser rangefinder is equal to 2, the pixel-to-actual distance conversion ratio factor is calculated based on the pixel distance of the two key points and the actual distance; the equation of the straight line where the pixels of the two key points are located is solved; the pixel distances of the remaining points on the curve profile to the straight line are calculated; the actual distances of the remaining points on the fire front curve profile and the straight line are calculated based on the pixel distance-to-actual distance conversion ratio factor; the actual positions of the remaining points on the fire front curve profile are calculated based on the actual positions of the points on the straight line and the actual distances of the remaining points on the fire front curve profile from the straight line; If the number of continuously detectable points P of the laser rangefinder is greater than or equal to 3, then for each point on the fire front curve contour, the three most adjacent key points are selected in turn to establish a local plane; through the perspective projection relationship between the pixel plane and the local plane, the two-dimensional pixel coordinates of the remaining points on the fire front curve contour are mapped to the three-dimensional space, so as to determine the actual positions of the remaining points on the fire front curve contour; predicting the spread of fire based on the location of the fire front and other information; The firefighting force retreat route is planned and dynamically updated in real time based on the position of the fire front and the spread prediction results.

2. According to claim 1, a forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod is characterized in that: The first processing of the frame image to detect whether there is a fire in the current frame includes: the frame image includes a visible light image and an infrared image, and features of different scales are extracted from the visible light image and the infrared image respectively; the two features are weightedly fused to obtain fused features; the fused features of all scales are spliced ​​to obtain a fused feature map of the fused multimodal image; the fused feature map is input into a fire detection model based on deep learning to perform fire detection, and detect and determine whether there is a fire in the current frame.

3. The forest fire fighting operation auxiliary decision-making method based on the airborne optoelectronic pod according to claim 2 is characterized in that: The second processing of the image frames in which the fire is detected includes: performing image segmentation on the image frames in which the fire is detected to obtain a binary image; using the Canny algorithm to extract the edge of the burning area in the binary image; calculating the optical flow between adjacent frames to obtain the optical flow speed; and identifying the curve formed by pixels whose optical flow speed is greater than a preset speed threshold as a fire front contour curve.

4. The forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod according to claim 3 is characterized in that: The third processing based on the fire front morphological characteristics to obtain the key points of the fire front contour curve includes: calculating the curvature value of each point along the fire front curve contour, sorting all the curvature values, and assigning an index i to each point; wherein the point with the largest curvature value is assigned an index value of 1, and the point with the smallest curvature value is assigned an index value of n, where n represents the number of fire points on the fire front curve contour; assuming that the number of continuously detectable points of the laser rangefinder on the airborne optoelectronic pod is P, the points on the fire front curve contour with index values ​​less than or equal to P are the key points of the fire front contour curve to be measured by the laser rangefinder.

5. The forest fire fighting operation auxiliary decision-making method based on the airborne optoelectronic pod according to claim 4 is characterized in that: The fourth processing of the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align with each key point includes: converting the relative pixel coordinates into normalized camera coordinates; calculating the angular deviations in the horizontal and vertical directions in the normalized camera coordinate system; and obtaining the servo system control amount according to the following formula based on the angular deviation: Da x =k x ·i x ,Yes y =k y ·i y In the formula, Δα x Indicates the adjustment angle of the servo system in the horizontal direction, Δα y Indicates the adjustment angle of the servo system in the vertical direction, k x and k y is the gain coefficient of the servo system; θ x ,θ y Represents the angular deviation in the horizontal and vertical directions respectively.

6. The forest fire fighting operation auxiliary decision-making method based on the airborne optoelectronic pod according to claim 5 is characterized in that: The real-time planning and dynamic updating of the firefighting force retreat route according to the position of the fire front and the spread prediction results include: calculating the evacuation capacity index ECI of each grid in the grid map of the area based on the terrain characteristics of the area around the fire scene; calculating the comprehensive safety index CSI of each grid based on the current fire front position and the spread prediction results; establishing a potential safety path evaluation function in combination with the evacuation capacity index ECI and the comprehensive safety index CSI; taking the maximization of the potential safety path evaluation function as the objective function, using the A* algorithm to solve and obtain the retreat route; combining the real-time fire front position and fire spread situation of the optoelectronic pod, dynamically updating the evacuation capacity index ECI and the comprehensive safety index CSI of each grid, and adjusting the retreat route in real time.

7. The forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod according to claim 6 is characterized in that: The calculation formula of the evacuation capacity index ECI is: Among them, S(i,j) represents the slope of grid (i,j); D(i,j) represents the obstacle density in the grid; α1 and α2 represent the adjustment coefficients; The calculation formula of the comprehensive safety index CSI is: CSI(i,j)=ω1·TSI(i,j)+ω2·SSI(i,j) Among them, TSI stands for time safety index, T safe (i,j) represents the safe time before the fire spreads to grid (i,j), T arrival It indicates the time required for fire fighting forces to reach the grid; SSI indicates the spatial safety index. d fire (i, j) represents the distance from grid (i, j) to the fire front, and ω1 and ω2 are weights; The expression of the potential safety path evaluation function is: PathScore(i,j)=λ1·ECI(i,j)+λ2·CSI(i,j) Among them, λ1 and λ2 represent the weights of the evacuation capacity index and the comprehensive safety index.

8. The forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod according to claim 1 is characterized in that: The other information includes: local weather information of the fire scene, topographic information, and combustible material property information.

9. A forest fire fighting operation auxiliary decision system based on an airborne optoelectronic pod, characterized in that: Based on the method described in any one of claims 1 to 8, the system comprises: A fire detection module is configured to collect continuous frame images and perform a first processing on the frame images to detect whether there is a fire in the current frame; A fire front identification module is configured to perform a second processing on the image frame where the fire is detected to identify the fire front; wherein the fire front is defined as the portion where the flame extends forward, and its contour curve is the boundary between the burning area and the unburned area; The fire front positioning module is configured to perform a third processing based on the morphological features of the fire front to obtain key points of the fire front contour curve; calculate the position of the key points relative to the center of the image plane of the frame image to obtain relative pixel coordinates; perform a fourth processing on the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align each key point, and perform distance measurement on each key point based on the servo system control amount; perform a fifth processing based on the relative position of each key point obtained by the distance measurement and the laser rangefinder to calculate the position of the fire front; a spread prediction module configured to predict the spread of fire based on the position of the fire front and other information; The safety path planning module is configured to plan and dynamically update the firefighting force retreat route in real time according to the position of the fire front and the spread prediction results.

Images