Forest fire fighting operation auxiliary decision-making method and system based on airborne photoelectric pod

Abstract

The invention discloses a forest fire fighting operation auxiliary decision-making method and system based on an airborne photoelectric pod, and relates to the technical field of forest fire prevention and control. The method is characterized by comprising the steps of collecting continuous frame images, and detecting whether a fire exists in a current frame or not; identifying the fire front of the image frame of which the fire behavior is detected; fire front profile curve key points are calculated and obtained based on the fire front morphological characteristics; calculating the position of the key point relative to the center of the image surface of the frame image; determining a servo system control quantity required for aligning the laser range finder with the key point according to the position; carrying out distance measurement on the key points based on the servo system control quantity; based on the relative position obtained through distance measurement, the fire front position is obtained through calculation; fire spreading is predicted according to the fire front position and other information; and performing real-time planning and dynamic updating on the fire power withdrawing route according to the fire front position and the spreading prediction result. According to the invention, commanders can be helped to quickly and accurately master the dynamic state of the fire scene, so that the forest fire emergency response efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of forest fire prevention and control, and in particular to a forest fire fighting operation auxiliary decision-making method and system based on an airborne optoelectronic pod. Background Art

[0002] Traditional forest fire monitoring methods mainly rely on ground stations and satellite remote sensing. These methods are insufficient in spatial resolution and real-time performance, making it difficult to provide detailed fire situation information, resulting in low efficiency of firefighting operations and even missing the best time to put out fires. Compared with ground stations and satellite remote sensing, airborne platforms are highly flexible and can quickly cover a large area of ​​fires and adjust the observation angle and range according to needs.

[0003] Airborne optoelectronic pods equipped with optical sensors, infrared sensors, laser rangefinders, servo control systems, and combined inertial navigation systems are very suitable for forest fire monitoring. For example, abnormal hot spots in the monitoring area can be roughly located by combining laser rangefinder information and optoelectronic pod posture information. However, the current fire monitoring methods based on airborne monitoring pods have a low degree of automation, often reducing the three-dimensional surface features of fire spread into one-dimensional point features, and have not formed an efficient forest fire fighting operation auxiliary decision-making system. Summary of the invention

[0004] In view of the above problems, the present invention proposes a forest fire fighting operation auxiliary decision-making method and system based on an airborne optoelectronic pod.

[0005] According to one aspect of the present invention, a forest fire fighting operation auxiliary decision method based on an airborne optoelectronic pod is proposed, the method comprising the following steps:

[0006] Collect continuous frame images and perform the first processing on the frame images to detect whether there is a fire in the current frame;

[0007] Processing the image frame where the fire is detected for the second time to identify the fire front; wherein the fire front is defined as the portion of the flame extending forward, and its contour curve is the boundary between the burning area and the unburned area;

[0008] The third processing is performed based on the morphological characteristics of the fire front to obtain the key points of the fire front contour curve;

[0009] Calculate the position of the key point relative to the center of the image plane of the frame image to obtain the relative pixel coordinates;

[0010] Processing the relative pixel coordinates for a fourth time to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align the laser rangefinder with each key point, and measuring the distance of each key point based on the servo system control amount;

[0011] The fifth processing is performed based on the relative positions of each key point obtained by ranging and the laser rangefinder to calculate the position of the fire front;

[0012] predicting the spread of fire based on the location of the fire front and other information;

[0013] The firefighting force retreat route is planned and dynamically updated in real time based on the position of the fire front and the spread prediction results.

[0014] Furthermore, the first processing of the frame image to detect whether there is a fire in the current frame includes: the frame image includes a visible light image and an infrared image, and features of different scales are extracted from the visible light image and the infrared image respectively; the two features are weightedly fused to obtain fused features; the fused features of all scales are spliced ​​to obtain a fused feature map of the fused multimodal image; the fused feature map is input into a fire detection model based on deep learning to perform fire detection, and detect and determine whether there is a fire in the current frame.

[0015] Furthermore, the second processing of the image frames in which the fire is detected includes: performing image segmentation on the image frames in which the fire is detected to obtain a binary image; using the Canny algorithm to extract the edge of the burning area in the binary image; calculating the optical flow between adjacent frames to obtain the optical flow speed; and identifying the curve formed by pixels whose optical flow speed is greater than a preset speed threshold as a fire front contour curve.

[0016] Furthermore, the third processing based on the morphological characteristics of the fire front to obtain the key points of the fire front contour curve includes: calculating the curvature value of each point along the fire front curve contour, sorting all curvature values, and assigning an index i to each point; wherein the point with the largest curvature value is assigned an index value of 1, and the point with the smallest curvature value is assigned an index value of n, where n represents the number of fire points on the fire front curve contour; assuming that the number of continuously detectable points of the laser rangefinder on the airborne optoelectronic pod is P, the points on the fire front curve contour with index values ​​less than or equal to P are the key points of the fire front contour curve to be measured by the laser rangefinder.

[0017] Furthermore, the fourth processing of the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align each key point includes: converting the relative pixel coordinates into normalized camera coordinates; calculating the angular deviations in the horizontal and vertical directions in the normalized camera coordinate system; and obtaining the servo system control amount according to the following formula based on the angular deviation:

[0018] Δα x =k x ·θ x , Δα y =k y ·θ y

[0019] In the formula, Δα x Indicates the adjustment angle of the servo system in the horizontal direction, Δα y Indicates the adjustment angle of the servo system in the vertical direction, k x and k y is the gain coefficient of the servo system; θ x ,θ y Represents the angular deviation in the horizontal and vertical directions respectively.

[0020] Furthermore, the fifth processing of the relative positions of each key point obtained based on the distance measurement and the laser rangefinder to calculate the position of the fire front includes:

[0021] If the number of continuously detectable points P of the laser rangefinder is equal to 1, the pixel sampling distance is calculated by the distance from the laser rangefinder to the ground, the pixel size and the focal length. The key point of the fire front is taken as the reference point, and the actual relative position of each pixel and the reference point is calculated by the pixel distance between the remaining pixels and the key point and the pixel sampling distance; the actual position of the remaining points on the curve contour of the fire front is calculated in combination with the actual coordinates of the key point;

[0022] If the number of continuously detectable points P of the laser rangefinder is equal to 2, the pixel-to-actual distance conversion ratio factor is calculated based on the pixel distance of the two key points and the actual distance; the equation of the straight line where the pixels of the two key points are located is solved; the pixel distances of the remaining points on the curve profile to the straight line are calculated; the actual distances of the remaining points on the fire front curve profile and the straight line are calculated based on the pixel distance-to-actual distance conversion ratio factor; the actual positions of the remaining points on the fire front curve profile are calculated based on the actual positions of the points on the straight line and the actual distances of the remaining points on the fire front curve profile from the straight line;

[0023] If the number of continuously detectable points P of the laser rangefinder is greater than or equal to 3, then for each point on the fire front curve contour, the three most adjacent key points are selected in turn to establish a local plane; through the perspective projection relationship between the pixel plane and the local plane, the two-dimensional pixel coordinates of the remaining points on the fire front curve contour are mapped to the three-dimensional space, thereby determining the actual positions of the remaining points on the fire front curve contour.

[0024] Furthermore, the real-time planning and dynamic updating of the firefighting force retreat route according to the position of the fire front and the spread prediction results include: calculating the evacuation capacity index ECI of each grid in the grid map of the area based on the terrain characteristics of the area around the fire scene; calculating the comprehensive safety index CSI of each grid based on the current fire front position and the spread prediction results; establishing a potential safe path evaluation function by combining the evacuation capacity index ECI and the comprehensive safety index CSI; taking the maximization of the potential safe path evaluation function as the objective function, using the A* algorithm to solve and obtain the retreat route; combining the real-time fire front position and fire spread situation of the optoelectronic pod, dynamically updating the evacuation capacity index ECI and the comprehensive safety index CSI of each grid, and adjusting the retreat route in real time.

[0025] Furthermore, the calculation formula of the evacuation capacity index ECI is:

[0026]

[0027] Among them, S(i,j) represents the slope of grid (i,j); D(i,j) represents the obstacle density in the grid; α1 and α2 represent the adjustment coefficients;

[0028] The calculation formula of the comprehensive safety index CSI is:

[0029] CSI(i,j)=ω1·TSI(i,j)+ω2·SSI(i,j)

[0030] Among them, TSI stands for time safety index, T safe (i,j) represents the safe time before the fire spreads to grid (i,j), T arrival It indicates the time required for fire fighting forces to reach the grid; SSI indicates the spatial safety index. d fire (i,j) represents the distance from grid (i,j) to the fire front, and ω1 and ω2 are weights.

[0031] Furthermore, the expression of the potential safety path evaluation function is:

[0032] PathScore(i,j)=λ1·ECI(i,j)+λ2·CSI(i,j)

[0033] Among them, λ1 and λ2 represent the weights of the evacuation capacity index and the comprehensive safety index.

[0034] Furthermore, the other information includes: local weather information of the fire scene, topographic information, and combustible material property information.

[0035] According to another aspect of the present invention, a forest fire fighting operation auxiliary decision system based on an airborne optoelectronic pod is proposed, the system comprising:

[0036] A fire detection module is configured to collect continuous frame images and perform a first processing on the frame images to detect whether there is a fire in the current frame;

[0037] A fire front identification module is configured to perform a second processing on the image frame where the fire is detected to identify the fire front; wherein the fire front is defined as the portion where the flame extends forward, and its contour curve is the boundary between the burning area and the unburned area;

[0038] The fire front positioning module is configured to perform a third processing based on the morphological features of the fire front to obtain key points of the fire front contour curve; calculate the position of the key points relative to the center of the image plane of the frame image to obtain relative pixel coordinates; perform a fourth processing on the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align each key point, and perform distance measurement on each key point based on the servo system control amount; perform a fifth processing based on the relative position of each key point obtained by the distance measurement and the laser rangefinder to calculate the position of the fire front;

[0039] a spread prediction module configured to predict the spread of fire based on the position of the fire front and other information;

[0040] The safety path planning module is configured to plan and dynamically update the firefighting force retreat route in real time according to the position of the fire front and the spread prediction results.

[0041] The beneficial technical effects of the present invention are:

[0042] The present invention proposes a forest fire fighting operation auxiliary decision-making method and system based on an airborne optoelectronic pod, which aims to help command personnel quickly and accurately grasp the dynamics of the fire scene and formulate effective firefighting strategies through efficient data collection, processing and fusion, thereby reducing the losses caused by the fire and improving the efficiency and effectiveness of forest fire emergency response. First, continuous frame images are collected and the frame images are processed for the first time to detect whether there is a fire in the current frame; fire detection is performed by fusing multimodal image data of the fire scene, which reduces the false alarm rate and missed alarm rate of fire detection; then, the image frame where the fire is detected is processed for the second time to identify the fire front; the third processing is performed based on the morphological characteristics of the fire front to obtain the key points of the fire front contour curve, and the position of the key points relative to the center of the image plane of the frame image is calculated to obtain the relative pixel coordinates; the relative pixel coordinates are processed for the fourth time to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align with each key point, and each key point is aligned based on the servo system control amount. Distance measurement; a fifth processing is performed based on the relative position of each key point obtained by the distance measurement and the laser rangefinder to calculate the position of the fire front; the perspective projection method based on the adaptive selection of the key points of the fire front can quickly obtain the three-dimensional coordinates of the fire front in one imaging, realize the global positioning of the fire front, and break through the limitation of the single-point positioning of the laser; then, the spread of the fire is predicted according to the position of the fire front and other information; the retreat route of the firefighting force is planned and dynamically updated in real time according to the position of the fire front and the spread prediction results; the above-mentioned fusion of multi-source data and intelligent algorithms provides an optimized path planning solution for the operation and retreat of the firefighting force, ensuring the safety of the firefighting operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification, and are used to further illustrate the preferred embodiments of the present invention and explain the principles and advantages of the present invention.

[0044] Figure 1 It is a flow chart of a forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod according to an embodiment of the present invention;

[0045] Figure 2 Another flow chart of a forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod according to an embodiment of the present invention;

[0046] Figure 3 This is an example diagram of the processing results of image segmentation, edge detection and fire front identification in an embodiment of the present invention.

[0047] Figure 4It is a schematic diagram of positioning the local fire front through the projection relationship between the pixel plane and the local plane in an embodiment of the present invention.

[0048] Figure 5 This is a flow chart of a spread prediction model that can implement real-time calibration of input parameters in an embodiment of the present invention.

[0049] Figure 6 This is an example diagram of the fire spread situation and evacuation route planning in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the scheme of the present invention, exemplary implementations or embodiments of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described implementations or embodiments are only implementations or embodiments of a part of the present invention, not all of them. Based on the implementations or embodiments of the present invention, all other implementations or embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0051] The embodiment of the present invention provides a forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod, such as Figures 1-2 As shown, the following steps are included:

[0052] S1. Collect continuous frame images and process the frame images for the first time to detect whether there is a fire in the current frame. The specific steps include: the frame images include visible light images and infrared images, and features of different scales are extracted from the visible light images and infrared images respectively; the two features are weightedly fused to obtain fused features; the fused features of all scales are spliced ​​to obtain a fused feature map of the fused multimodal image; the fused feature map is input into a fire detection model based on deep learning to perform fire detection, and detect whether there is a fire in the current frame;

[0053] S2. Processing the image frame where the fire is detected for the second time to identify the fire front; wherein the fire front is defined as the part where the flame extends forward, and its contour curve is the boundary between the burning area and the unburned area; the specific steps include: performing image segmentation on the image frame where the fire is detected to obtain a binary image; using the Canny algorithm to extract the edge of the burning area in the binary image; calculating the optical flow between adjacent frames, and then obtaining the optical flow speed; identifying the curve formed by pixels whose optical flow speed is greater than a preset speed threshold as the fire front contour curve;

[0054] S3, perform a third processing based on the fire front morphological characteristics to obtain the key points of the fire front contour curve; the specific steps include: calculating the curvature value of each point along the fire front curve contour, sorting all the curvature values, and assigning an index i to each point; wherein the point with the largest curvature value is assigned an index value of 1, and the point with the smallest curvature value is assigned an index value of n, where n represents the number of fire points on the fire front curve contour; assuming that the number of continuously detectable points of the laser rangefinder on the airborne optoelectronic pod is P, the points on the fire front curve contour with index values ​​less than or equal to P are the key points of the fire front contour curve to be measured by the laser rangefinder;

[0055] S4, calculating the position of the key point relative to the center of the image plane of the frame image to obtain the relative pixel coordinates; performing a fourth processing on the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align with each key point, and measuring the distance of each key point based on the servo system control amount; the specific steps include: converting the relative pixel coordinates into normalized camera coordinates; calculating the angular deviations in the horizontal and vertical directions in the normalized camera coordinate system; and obtaining the servo system control amount according to the angular deviation calculation;

[0056] S5, based on the relative position of each key point obtained by ranging and the laser rangefinder, a fifth processing is performed to calculate the position of the fire front; the specific steps include: if the number of continuously detectable points P of the laser rangefinder is equal to 1, the pixel sampling distance is calculated by the distance from the laser rangefinder to the ground, the pixel size and the focal length, the fire front key point is used as a reference point, and the actual relative position of each pixel and the reference point is calculated by the pixel distance between the remaining pixels and the key point and the pixel sampling distance; the actual position of the remaining points on the curve contour of the fire front is calculated in combination with the actual coordinates of the key point; if the number of continuously detectable points P of the laser rangefinder is equal to 2, the pixel and actual distance conversion ratio factor is calculated based on the pixel distance and actual distance of the two key points; the square of the straight line where the pixels of the two key points are located is solved. program; calculate the pixel distances of the remaining points on the curve profile to the straight line; calculate the actual distances of the remaining points on the fire front curve profile and the straight line according to the conversion ratio factor of the pixel distance and the actual distance; calculate the actual positions of the remaining points on the fire front curve profile according to the actual positions of the points on the straight line and the actual distances of the remaining points on the fire front curve profile from the straight line; if the number of continuously detectable points P of the laser rangefinder is greater than or equal to 3, then for each point on the fire front curve profile, select the three most adjacent key points in turn to establish a local plane; through the perspective projection relationship between the pixel plane and the local plane, map the two-dimensional pixel coordinates of the remaining points on the fire front curve profile to the three-dimensional space, so as to determine the actual positions of the remaining points on the fire front curve profile;

[0057] S6. predicting the spread of fire according to the position of the fire front and other information;

[0058] S7. According to the position of the fire front and the fire spread prediction results, the retreat route of the fire fighting force is planned and dynamically updated in real time; the specific steps include: calculating the evacuation capacity index of each grid in the grid map of the area based on the terrain characteristics of the area around the fire scene; calculating the comprehensive safety index of each grid based on the current fire front position and the spread prediction results; establishing a potential safe path evaluation function in combination with the evacuation capacity index and the comprehensive safety index; taking the maximization of the potential safe path evaluation function as the objective function, using the A* algorithm to solve and obtain the retreat route; combining the real-time fire front position and fire spread situation of the optoelectronic pod, dynamically updating the evacuation capacity index and the comprehensive safety index of each grid, and adjusting the retreat route in real time.

[0059] The method starts from S1. In S1, continuous frame images are collected and the frame images are processed for the first time to detect whether there is a fire in the current frame.

[0060] According to an embodiment of the present invention, the visible light and infrared images acquired by the optoelectronic pod are firstly used as input, the features of the multimodal images are fused, and the presence of a fire in the current field of view is detected in real time based on the fused images. It should be noted that the sensors carried by the airborne optoelectronic pod should include visible light cameras, infrared cameras, laser rangefinders, inertial navigation systems, etc.

[0061] As an example, the specific method of visible light and infrared image fusion is:

[0062] S11. Use convolutional neural network CNN to process infrared image and visible light image I RGB , using n convolutional layers L1, L2, ... L with different sizes of convolution kernels n ; The output feature map of each convolutional layer is: in, and They are the feature maps of infrared and visible light images at the i-th convolutional layer respectively;

[0063] S12, use bilinear interpolation to adjust the feature map to the same size H×W;

[0064] S13. For each scale i, the infrared feature map and the visible light feature map are fused using a weighted summation method:

[0065]

[0066] S14. Concatenate the fused feature maps of all scales to obtain the final fused feature map:

[0067]

[0068] As an example, the fire detection model based on deep learning adopts a lightweight deep learning model with MobileNetV3 as the backbone network. The specific structure is: the input data first passes through a standard convolutional layer; then it is connected to n depth-separable convolution blocks, each of which includes deep convolution and point-by-point convolution, followed by Batch Normalization and Hard Swish; global average pooling is used to reduce the spatial dimension of the feature map to 1 to obtain a feature vector of fixed length; m fully connected layers are used to further process the extracted features; the output layer uses a Sigmoid activation function to output the fire detection results in a binary classification manner.

[0069] Then S2 is executed, in which the image frame in which the fire is detected is processed for the second time to identify the fire front.

[0070] According to an embodiment of the present invention, for an image frame in which a fire is detected, the burning area is segmented, its contour is extracted, and the fire front and the fire back front are identified; wherein the fire front is the part where the flame extends forward and the boundary between the burning area and the unburned area, and the fire back front is the boundary between the burning area and the burned area.

[0071] As an example, the specific steps of burning area segmentation, contour extraction and fire front identification include:

[0072] S21, custom threshold T for image segmentation:

[0073]

[0074] Among them, p(i) is the gray probability of the pixel, u1 and u2 are the average values ​​before and after the segmentation point, i is the gray level, and L is the maximum gray level; a binary image is output, in which the white area represents the detected burning area, and the black area represents the burned area and the unburned area;

[0075] S22, using the Canny algorithm to extract the edge of the burning area;

[0076] S23. Use the Lucas-Kanade method to calculate the optical flow between adjacent frames I(t) and I(t+1). The optical flow equation is:

[0077] I x u+I y v+I t =0

[0078] Among them, I(x) and I(y) are the spatial gradients of the image, and I(t) is the temporal gradient. By solving the optical flow equation, the optical flow component (u, v) of each pixel is obtained;

[0079] S24. Calculate the optical flow velocity:

[0080]

[0081] S25, optical flow speed is greater than speed threshold V th The curve formed by the pixels is the fire front. Figure 3 The fire front identification process and results are shown.

[0082] Then, S3 is executed. In S3, a third processing is performed based on the morphological features of the fire front to obtain key points of the fire front contour curve.

[0083] According to an embodiment of the present invention, the curvature of each point is calculated along the fire front contour, and the curvature values ​​are sorted according to the calculated curvature values, and an index i is assigned to each point. The point with the largest curvature value is assigned an index value of 1, and the point with the smallest curvature value is assigned an index value of n, which is the number of fire points on the fire front contour. Assuming that the number of points that the optoelectronic pod laser rangefinder can continuously detect is P, the points on the fire front contour with index values ​​less than or equal to P are the key points of the fire front to be detected by the laser rangefinder. It should be noted that the number of key points of the fire front that the laser rangefinder needs to detect is determined by the hardware properties of the laser rangefinder.

[0084] As an example, the curvature calculation method of each pixel point on the fire front contour is:

[0085]

[0086] Among them, x′ and y′ are the reciprocals of the current point, and x″ and y″ are the second-order derivatives of the current point; the greater the curvature, the higher the degree of curvature at the point.

[0087] Then, S4 is executed, in which the position of the key point relative to the center of the image plane of the frame image is calculated to obtain the relative pixel coordinates; the relative pixel coordinates are processed for the fourth time to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align with each key point, and the distance of each key point is measured based on the servo system control amount.

[0088] According to an embodiment of the present invention, after extracting key points based on the morphological features of the fire front, the position of the key point with index P relative to the center of the image plane is calculated, the servo system control amount required for the laser rangefinder to align the point is determined, the optoelectronic pod servo system is controlled to achieve automatic alignment and ranging, and key points with index values ​​of P-1, P-2, etc. are detected in sequence until all ranging is completed. Ensure that high-priority key points with large curvature can be detected with minimal time delay and the next step can be performed quickly.

[0089] As an example, the specific steps of calculating the position of a key point with index P relative to the center of the image plane and determining the servo system control amount required for the laser rangefinder to align the point include:

[0090] S41, calculating the position of the key point with index P relative to the center of the image plane, and obtaining the relative coordinates of the pixel;

[0091] S42, convert the pixel relative coordinates (u, v) to normalized camera coordinates (x n ,y n ):

[0092]

[0093] Among them, f x is the focal length of the camera in the horizontal direction; f y is the focal length of the camera in the vertical direction;

[0094] S43. Calculate the angular deviation θ in the horizontal and vertical directions in the normalized camera coordinate system. x ,θ y :

[0095] θ x =arctan(x n ), θ y =arctan(y n );

[0096] S44, the calculated angle deviation θ x and θ y Converted into the control quantity of the servo system:

[0097] Δα x =k x ·θ x , Δα y =k y ·θ y

[0098] Among them, Δα x is the adjustment angle of the servo system in the horizontal direction, Δα y is the adjustment angle of the servo system in the vertical direction, k x and k y is the gain factor of the servo system.

[0099] Then, S5 is executed. In S5, a fifth processing is performed based on the relative positions of each key point obtained by ranging and the laser rangefinder to calculate and obtain the position of the fire front.

[0100] According to an embodiment of the present invention, a specific solution for determining the location of the fire front according to the number of continuously detectable points P of the laser rangefinder includes:

[0101] S51. If P=1, the pixel sampling distance is calculated by the distance from the camera to the ground, the pixel size and the focal length. The key point of the fire front with an index value of 1 is used as the reference point. The relative position of each pixel and the reference point is calculated by the pixel distance between the remaining pixels and the key point and the pixel sampling distance. The fire front is positioned by combining its own absolute coordinates measured by the inertial navigation system, the external parameters of the laser and the inertial navigation system, the relative position of the key point and the laser, and the relative position of the remaining pixel points on the fire front and the key point.

[0102] As an example, the specific steps of positioning the fire front corresponding to P=1 in S51 include:

[0103] S511, based on the distance H from the camera to the ground and the focal length f of the camera in the horizontal direction x , vertical focal length f y , calculate the pixel sampling distance:

[0104]

[0105] Among them, d x is the actual distance corresponding to each pixel in the horizontal direction, d y is the actual distance corresponding to each pixel in the vertical direction, s x and y are the pixel sizes in the horizontal and vertical directions, respectively;

[0106] S512, calculate other pixel points (x i ,y i ) and the actual relative position of the key point (X1, Y1, Z1):

[0107] ΔX i =(x i -x1)·d x ,ΔY i =(y i -y1)·d y

[0108] S513. Calculate the actual ground coordinates of the remaining points based on the actual coordinates of the key points:

[0109] X i =X1+ΔX i ,Y i =Y1+ΔY i

[0110] S514, repeat S512 to S513 to calculate each pixel point on the fire front to obtain the actual position of the remaining points.

[0111] S52, if P=2, obtain the conversion relationship between the two key points through the actual distance and pixel distance between the two key points; take any key point as the reference point, combine the conversion relationship between pixel distance and actual distance to obtain the relative position of each pixel point and the key point on the straight line where the two key points are located, and then finally obtain the relative position of each pixel point and the key point on the fire front according to the distance between the remaining pixels on the fire front and the straight line where the two key points are located; combine the absolute coordinates of the inertial navigation system measured by itself, the external parameters of the laser and the inertial navigation system, the relative position of the key point and the laser, and the relative position of the remaining pixel points and the key point on the fire front to realize the positioning of the fire front.

[0112] As an example, the specific steps of positioning the fire front corresponding to P=2 in S52 include:

[0113] S521, calculating the pixel-to-actual distance conversion ratio factor α based on the pixel distance and the actual distance of the two key points:

[0114]

[0115] Among them, (x1, y1) and (x2, y2) are the pixel coordinates of the two key points respectively, and (X1, Y1) and (X2, Y2) are the actual ground coordinates of the key points actually measured through pose transformation;

[0116] S522, solving the equation of the straight line where the two key point pixels are located:

[0117] Ax+By+V=0

[0118] Among them, A, B, C are the parameters to be found;

[0119] S523, calculate other pixels (x i ,y i ) to the straight line in pixels:

[0120]

[0121] S524, calculate the actual distance D between any pixel on the fire front and the straight line according to the conversion ratio factor α between the pixel distance and the actual distance:

[0122] D=α·d

[0123] S525. Calculate the actual ground coordinates of each point on the fire front according to the actual ground coordinates of each point on the straight line and the actual distances of each point on the fire front from the straight line.

[0124] S53, if P>=3, for each pixel point on the fire front, select the three most adjacent key points in turn to establish a local plane, and then, through the perspective projection relationship between the pixel plane and the local plane, map the two-dimensional pixel coordinates of the remaining points on the fire front to the three-dimensional space, and finally realize the positioning of the fire front.

[0125] As an example, the specific steps of positioning the fire front corresponding to P>=3 in S53 include:

[0126] S531. Select three key points (X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3) on the fire front to define the local plane, and calculate the plane normal vector n by vector cross product:

[0127]

[0128] S532. Using the normal vector n = (a, b, c) and a point (X1, Y1, Z1) on the plane, the plane equation can be expressed as:

[0129] ax+by+cz+d=0

[0130] Among them, d is a constant term, which can be obtained by substituting a point on the plane into:

[0131] d=-(aX1+bY1+cZ1)

[0132] S533, converting the world coordinates into camera coordinates through the camera intrinsic parameter matrix K and the extrinsic parameter matrix (including the rotation matrix R and the translation vector T);

[0133] S534. For the pixel coordinates (u, v) of the non-keypoint, the inverse transformation of the projection matrix is ​​solved to map it from the two-dimensional pixel plane back to the three-dimensional space:

[0134]

[0135] Among them, λ is the scale factor, and (X, Y, Z) is obtained by the inverse projection relationship;

[0136] S535. The solved three-dimensional coordinates (X, Y, Z) may not be completely on the local plane. It is necessary to project them back to the local plane, that is, find the nearest point (X p ,Y p ,Z p )satisfy:

[0137] ax+by+cz+d=0

[0138] (X p ,Y p ,Z p ) is the final three-dimensional coordinate of the fire front pixel.

[0139] Figure 4 A local fire front positioning diagram is shown using the projection relationship between the pixel plane and the local plane.

[0140] Then S6 is executed, in which the fire spread is predicted based on the position of the fire front and other information.

[0141] According to an embodiment of the present invention, the spread prediction can be performed based on the fire front position and the local meteorological information, topographic information, and combustible property information of the fire scene, and the position of the fire line after time δt can be obtained. The spread prediction method can update the spread prediction results through real-time data, and calibrate the input parameters of the spread prediction by minimizing the real-time observation error of the fire front, including but not limited to the moisture content of the combustibles and the high-resolution wind field. Among them, the local meteorological information of the fire scene can be crawled through relevant authoritative websites, or collected in real time through an airborne multi-factor meteorological measurement device, or transmitted in real time with temporary meteorological stations around the fire scene, etc. There are differences in the spatiotemporal resolution of meteorological data obtained by different means.

[0142] As an example, the steps to predict the spread of a fire based on the location of the fire front and other information include:

[0143] S61. Construct a spreading prediction model combining the Rothermel spreading velocity model and the level set method (as a two-dimensional extended model);

[0144] S62, at time t0, input the fire front position observed at time t0, local real-time meteorological data of the fire scene, the collected fire scene slope, slope direction, altitude, combustible type, and combustible moisture content, and predict the fire front position after time t1;

[0145] S63. At time t1, the Hausdorff distance between the currently observed fire front position and the prediction result at time t0 is first calculated as the prediction error, and then the error is minimized by the differential evolution algorithm to achieve the calibration of the input parameters. Finally, the fire front position at time t2 is predicted based on the calibrated parameters; the prediction of the fire front position at subsequent moments is achieved according to the above process. The flow chart of the spread prediction method is shown in the figure. Figure 5 shown.

[0146] Finally, S7 is executed, in which the retreat route of the fire fighting force is planned and dynamically updated in real time according to the position of the fire front and the spread prediction result.

[0147] According to the embodiment of the present invention, the firefighting force's current position is taken as the starting point, based on the real-time fire front position information obtained by the optoelectronic pod and the spread situation deduction information obtained in step 6, the firefighting force's retreat route is planned and dynamically updated in real time. The specific steps include:

[0148] S71. Calculate the evacuation capacity index (ECI) of each grid based on the terrain characteristics (including slope, obstacles, and road conditions), and use the ECI to measure the passability of the grid:

[0149]

[0150] Among them, S(i,j) represents the slope of grid (i,j); D(i,j) represents the obstacle density in the grid; α1 and α2 represent the adjustment coefficients;

[0151] S72. Based on the current fire front position and spread prediction results, the comprehensive safety index (CSI) of each grid in the area is calculated. The comprehensive safety index includes two parts: the time safety index (TSI) and the space safety index (SSI):

[0152] CSI(i,j)=ω1·TSI(i,j)+ω2·SSI(i,j)

[0153] in, T safe (i,j) represents the safe time before the fire spreads to grid (i,j), T arrival Indicates the time required for fire fighting forces to reach the grid; d fire (i,j) represents the distance from grid (i,j) to the fire front, and ω1 and ω2 are weights.

[0154] S73. Combine the evacuation capacity index and the comprehensive safety index to establish a potential safety path evaluation function for the fire fighting force safe evacuation path planning algorithm:

[0155] PathScore(i,j)=λ1·ECI(i,j)+λ2·CSI(i,j)

[0156] Among them, λ1 and λ2 are the weights of evacuation capacity and safety index;

[0157] S74. Use the A* algorithm to balance the safety of the path and the evacuation efficiency, maximize the evaluation function PathScore(i,j), and generate the optimal evacuation path for the firefighting force:

[0158]

[0159] S75. Combined with the real-time fire front position and fire spread situation of the optoelectronic pod, dynamically update the evacuation capacity index and comprehensive safety index of each grid, and adjust the retreat path in real time. The updated retreat path meets the new safety requirements while maintaining the continuity and effectiveness of the path as much as possible. Figure 6 shown.

[0160] Another embodiment of the present invention provides a forest fire fighting operation auxiliary decision system based on an airborne optoelectronic pod, the system comprising:

[0161] A fire detection module is configured to collect continuous frame images and perform a first processing on the frame images to detect whether there is a fire in the current frame;

[0162] A fire front identification module is configured to perform a second processing on the image frame where the fire is detected to identify the fire front; wherein the fire front is defined as the portion where the flame extends forward, and its contour curve is the boundary between the burning area and the unburned area;

[0163] The fire front positioning module is configured to perform a third processing based on the morphological features of the fire front to obtain key points of the fire front contour curve; calculate the position of the key points relative to the center of the image plane of the frame image to obtain relative pixel coordinates; perform a fourth processing on the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align each key point, and perform distance measurement on each key point based on the servo system control amount; perform a fifth processing based on the relative position of each key point obtained by the distance measurement and the laser rangefinder to calculate the position of the fire front;

[0164] a spread prediction module configured to predict the spread of fire based on the position of the fire front and other information;

[0165] The safety path planning module is configured to plan and dynamically update the firefighting force retreat route in real time according to the position of the fire front and the spread prediction results.

[0166] The functions of the forest fire fighting operation auxiliary decision-making system based on an airborne optoelectronic pod described in this embodiment can be described by the aforementioned forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod. Therefore, for the parts not described in detail in this embodiment, please refer to the above method embodiments and will not be repeated here.

[0167] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments are contemplated within the scope of the invention thus described. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. A forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod, characterized in that: The following steps are involved: Collect continuous frame images and perform the first processing on the frame images to detect whether there is a fire in the current frame; Processing the image frame where the fire is detected for the second time to identify the fire front; wherein the fire front is defined as the portion of the flame extending forward, and its contour curve is the boundary between the burning area and the unburned area; The third processing is performed based on the morphological characteristics of the fire front to obtain the key points of the fire front contour curve; Calculate the position of the key point relative to the center of the image plane of the frame image to obtain the relative pixel coordinates; Processing the relative pixel coordinates for a fourth time to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align the laser rangefinder with each key point, and measuring the distance of each key point based on the servo system control amount; The fifth processing is performed based on the relative positions of each key point obtained by ranging and the laser rangefinder to calculate the position of the fire front; predicting the spread of fire based on the location of the fire front and other information; The firefighting force retreat route is planned and dynamically updated in real time based on the position of the fire front and the spread prediction results.

2. According to claim 1, a forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod is characterized in that: The first processing of the frame image to detect whether there is a fire in the current frame includes: the frame image includes a visible light image and an infrared image, and features of different scales are extracted from the visible light image and the infrared image respectively; the two features are weightedly fused to obtain fused features; the fused features of all scales are spliced ​​to obtain a fused feature map of the fused multimodal image; the fused feature map is input into a fire detection model based on deep learning to perform fire detection, and detect and determine whether there is a fire in the current frame.

3. The forest fire fighting operation auxiliary decision-making method based on the airborne optoelectronic pod according to claim 2 is characterized in that: The second processing of the image frames in which the fire is detected includes: performing image segmentation on the image frames in which the fire is detected to obtain a binary image; using the Canny algorithm to extract the edge of the burning area in the binary image; calculating the optical flow between adjacent frames to obtain the optical flow speed; and identifying the curve formed by pixels whose optical flow speed is greater than a preset speed threshold as a fire front contour curve.

4. The forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod according to claim 3 is characterized in that: The third processing based on the fire front morphological characteristics to obtain the key points of the fire front contour curve includes: calculating the curvature value of each point along the fire front curve contour, sorting all the curvature values, and assigning an index i to each point; wherein the point with the largest curvature value is assigned an index value of 1, and the point with the smallest curvature value is assigned an index value of n, where n represents the number of fire points on the fire front curve contour; assuming that the number of continuously detectable points of the laser rangefinder on the airborne optoelectronic pod is P, the points on the fire front curve contour with index values ​​less than or equal to P are the key points of the fire front contour curve to be measured by the laser rangefinder.

5. The forest fire fighting operation auxiliary decision-making method based on the airborne optoelectronic pod according to claim 4 is characterized in that: The fourth processing of the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align with each key point includes: converting the relative pixel coordinates into normalized camera coordinates; calculating the angular deviations in the horizontal and vertical directions in the normalized camera coordinate system; and obtaining the servo system control amount according to the following formula based on the angular deviation: Da x =k x ·i x ,Yes y =k y ·i y In the formula, Δα x Indicates the adjustment angle of the servo system in the horizontal direction, Δα y Indicates the adjustment angle of the servo system in the vertical direction, k x and k y is the gain coefficient of the servo system; θ x ,θ y Represents the angular deviation in the horizontal and vertical directions respectively.

6. The forest fire fighting operation auxiliary decision-making method based on the airborne optoelectronic pod according to claim 5 is characterized in that: The fifth processing based on the relative positions of each key point obtained by ranging and the laser rangefinder to calculate the position of the fire front includes: If the number of continuously detectable points P of the laser rangefinder is equal to 1, the pixel sampling distance is calculated by the distance from the laser rangefinder to the ground, the pixel size and the focal length. The key point of the fire front is taken as the reference point, and the actual relative position of each pixel and the reference point is calculated by the pixel distance between the remaining pixels and the key point and the pixel sampling distance; the actual position of the remaining points on the curve contour of the fire front is calculated in combination with the actual coordinates of the key point; If the number of continuously detectable points P of the laser rangefinder is equal to 2, the pixel-to-actual distance conversion ratio factor is calculated based on the pixel distance of the two key points and the actual distance; the equation of the straight line where the pixels of the two key points are located is solved; the pixel distances of the remaining points on the curve profile to the straight line are calculated; the actual distances of the remaining points on the fire front curve profile and the straight line are calculated based on the pixel distance-to-actual distance conversion ratio factor; the actual positions of the remaining points on the fire front curve profile are calculated based on the actual positions of the points on the straight line and the actual distances of the remaining points on the fire front curve profile from the straight line; If the number of continuously detectable points P of the laser rangefinder is greater than or equal to 3, then for each point on the fire front curve contour, the three most adjacent key points are selected in turn to establish a local plane; through the perspective projection relationship between the pixel plane and the local plane, the two-dimensional pixel coordinates of the remaining points on the fire front curve contour are mapped to the three-dimensional space, thereby determining the actual positions of the remaining points on the fire front curve contour.

7. The forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod according to claim 6 is characterized in that: The real-time planning and dynamic updating of the firefighting force retreat route according to the position of the fire front and the spread prediction results include: calculating the evacuation capacity index ECI of each grid in the grid map of the area based on the terrain characteristics of the area around the fire scene; calculating the comprehensive safety index CSI of each grid based on the current fire front position and the spread prediction results; establishing a potential safety path evaluation function in combination with the evacuation capacity index ECI and the comprehensive safety index CSI; taking the maximization of the potential safety path evaluation function as the objective function, using the A* algorithm to solve and obtain the retreat route; combining the real-time fire front position and fire spread situation of the optoelectronic pod, dynamically updating the evacuation capacity index ECI and the comprehensive safety index CSI of each grid, and adjusting the retreat route in real time.

8. The forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod according to claim 7 is characterized in that: The calculation formula of the evacuation capacity index ECI is: Among them, S(i,j) represents the slope of grid (i,j); D(i,j) represents the obstacle density in the grid; α1 and α2 represent the adjustment coefficients; The calculation formula of the comprehensive safety index CSI is: CSI(i,j)=ω1·TSI(i,j)+ω2·SSI(i,j) Among them, TSI stands for time safety index, T safe (i,j) represents the safe time before the fire spreads to grid (i,j), T arrival It indicates the time required for fire fighting forces to reach the grid; SSI indicates the spatial safety index. d fire (i, j) represents the distance from grid (i, j) to the fire front, and ω1 and ω2 are weights; The expression of the potential safety path evaluation function is: PathScore(i,j)=λ1·ECI(i,j)+λ2·CSI(i,j) Among them, λ1 and λ2 represent the weights of the evacuation capacity index and the comprehensive safety index.

9. The forest fire fighting operation auxiliary decision-making method based on an airborne optoelectronic pod according to claim 1 is characterized in that: The other information includes: local weather information of the fire scene, topographic information, and combustible material property information.

10. A forest fire fighting operation auxiliary decision system based on an airborne optoelectronic pod, characterized in that: include: A fire detection module is configured to collect continuous frame images and perform a first processing on the frame images to detect whether there is a fire in the current frame; A fire front identification module is configured to perform a second processing on the image frame where the fire is detected to identify the fire front; wherein the fire front is defined as the portion where the flame extends forward, and its contour curve is the boundary between the burning area and the unburned area; The fire front positioning module is configured to perform a third processing based on the morphological features of the fire front to obtain key points of the fire front contour curve; calculate the position of the key points relative to the center of the image plane of the frame image to obtain relative pixel coordinates; perform a fourth processing on the relative pixel coordinates to determine the servo system control amount required for the laser rangefinder on the airborne optoelectronic pod to align each key point, and perform distance measurement on each key point based on the servo system control amount; perform a fifth processing based on the relative position of each key point obtained by the distance measurement and the laser rangefinder to calculate the position of the fire front; a spread prediction module configured to predict the spread of fire based on the position of the fire front and other information; The safety path planning module is configured to plan and dynamically update the firefighting force retreat route in real time according to the position of the fire front and the spread prediction results.

Images