An auxiliary fire extinguishing method and system based on unmanned aerial vehicle information and image processing technology

Abstract

This invention discloses an assisted firefighting method and system based on UAV information and image processing technology. The method includes: step S1, using a UAV equipped with preset equipment to collect images of a fire scene; step S2, performing smoke removal preprocessing on the fire scene images to obtain a preprocessed image; step S3, performing target detection on the preprocessed image to obtain a target detection result; step S4, performing visual target localization at the fire scene based on the UAV's gimbal pose information and the target detection result; and step S5, using the target detection result and the visual target localization to assist in the scheduling of firefighting resources, thus completing the assisted firefighting based on UAV information and image processing technology. The improved YOLOv8 model is used to perform target detection on the preprocessed image. Based on the camera imaging principle, and combining the target detection result with the UAV's monocular gimbal sensor data, the fire source target is three-dimensionally located to assist in the scheduling of firefighting resources.

Description

Technical Field

[0001] This invention relates to the fields of fire rescue, firefighting, and image processing technology, specifically to an auxiliary firefighting method and system based on UAV information and image processing technology. Background Technology

[0002] Fires often cause severe casualties and property damage, having a significant impact on society and the economy. They also pose a significant threat to firefighters. Improving the intelligence level of fire and rescue operations and increasing their efficiency and safety are urgent issues to be addressed. In fire and rescue work, obtaining fire scene information safely and quickly is crucial. Drones, with their advantages of maneuverability, low cost, rich functionality, and strong scalability, have significant application value in firefighting scenarios. Drones can carry various sensors and equipment, such as data transmission and image transmission modules, to quickly reach the fire scene, communicate with the fire command center, and provide firefighters with real-time images of the fire. By integrating drone images and sensor information using image processing technology and intelligent algorithms, information such as the location of the fire source and the size of the fire can be obtained, thereby assisting firefighters in dispatching firefighting resources and conducting efficient and safe firefighting and disaster relief.

[0003] In firefighting scenarios, the higher flight altitude and wider field of view of drones due to flames, smoke, and water jets result in a higher proportion of small targets captured in the images. Furthermore, the images contain non-uniform light and smoke generated by flames. Traditional atmospheric scattering models default to pixel values ​​of (255, 255, 255) for aerosols, which are only suitable for scenes with uniform light intensity and therefore unsuitable for firefighting scenarios. Therefore, there is an urgent need to research a target detection scheme for drone images applicable to firefighting scenarios.

[0004] Traditional 3D visual localization methods mostly rely on binocular vision systems, using feature matching algorithms to generate disparity maps or point clouds, and then perform 3D localization of targets in images. However, compared to binocular vision systems, drones using monocular vision systems are lighter and cheaper. Therefore, this design uses a drone equipped with a monocular camera. In this case, traditional binocular vision localization methods fail, and there is an urgent need to research a 3D visual localization method for drones equipped with monocular cameras. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an assisted firefighting method based on UAV information and image processing technology, the method specifically comprising:

[0006] Step S1: Use a drone equipped with preset equipment to collect images of the fire scene;

[0007] Step S2: Perform smoke removal preprocessing on the fire scene image to obtain a preprocessed image;

[0008] Step S3: Perform target detection on the preprocessed image to obtain the target detection result;

[0009] Step S4: Based on the gimbal pose information of the UAV and the target detection results, perform visual target localization at the fire scene;

[0010] Step S5: Based on the target detection results and the visual target positioning, assist in scheduling fire-fighting resources to complete fire-fighting assistance based on UAV information and image processing technology.

[0011] Optionally, in step S2, the process of obtaining the preprocessed image specifically includes:

[0012] A neural network model is constructed based on UNet and ResNet networks. During neural network training, Berlin noise is used to generate model parameters t(x) and c(x). The model parameters t(x) and c(x) are input into an improved atmospheric scattering model to generate smoke images. The neural network model is trained using the smoke images and existing fire scene images to obtain a trained neural network model.

[0013] The fire scene image is input into a trained neural network model to obtain a preprocessed image.

[0014] Optionally, the process of constructing a neural network model based on UNet and ResNet specifically includes:

[0015] The neural network model includes an encoder network and a decoder network;

[0016] The encoder network uses a ResNet network to extract features. The ResNet network contains five neural network layers. The first layer consists of a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer. The input image is resized to a dimension of 640×640×3, and the output is a 320×320×3 feature map. The second, third, and fourth layers are residual network layers, which consist of convolutional layers and residual connections.

[0017] The decoder network consists of five neural network layers. The first layer receives the output of the encoder. The first, second, third, and fourth layers consist of a transposed convolutional layer, a channel concatenation layer, and a residual network layer. The input of each layer first passes through a transposed convolutional layer, then passes through a channel concatenation layer to concatenate the feature map of the same dimension as the feature map in the encoder network, and finally passes through a residual network layer to extract features and output them. The fifth layer consists of two convolutional layers, receiving a 640×640×32 feature map output from the fourth layer of the decoder network. The output dimension is 640×640×3, which is the decoder output.

[0018] Optionally, the process of generating smoke images by using the Berlin noise generation model parameters t(x) and c(x) and inputting the model parameters t(x) and c(x) into the improved atmospheric scattering model specifically includes:

[0019] The generation process of t(x) specifically includes:

[0020] The noise layer number was set to 6, persistence to 0.5, porosity to 2.0, and scale to 250. After normalization, a Burmester noise map P was generated. noise ;

[0021] Based on the aforementioned Berlin noise map, the smoke transmittance map t(x) is generated:

[0022] t(x)=P noise ·(t max -t min )+t min

[0023] Among them, t max t represents the maximum transmittance. min This represents the minimum transmittance.

[0024] The generation process of c(x) specifically includes:

[0025] Based on the aforementioned Berlin noise map, the RGB values ​​c(x) of the smoke and haze are generated:

[0026] c(x)=P noise ·(c max -c min )+c min ;

[0027] Among them, c max c represents the maximum RGB value of smoke and haze. max This represents the minimum RGB value for smoke and haze.

[0028] Substituting the t(x) and c(x) obtained above into the improved atmospheric scattering model, smoke is added to the clear image to obtain a smoke image.

[0029] Optionally, the improved atmospheric scattering model is specifically as follows:

[0030] I(x)=J(x)t(x)+c(x)(1-t(x))

[0031] Where I(x) is the fire scene image; J(x) is the image after ideal defogging processing; t(x) is the smoke transmittance map; and c(x) is the RGB value of the smoke and haze.

[0032] Optionally, step S4, the process of visual target localization at the fire scene, specifically includes:

[0033] Based on the camera calibration method, the camera intrinsic parameter matrix is ​​obtained, and distortion processing is performed on the camera based on the camera intrinsic parameter matrix;

[0034] The drone's location information is obtained using the orthodontic drone positioning module;

[0035] The attitude information of the drone gimbal camera is obtained based on the drone sensors after orthodontic treatment.

[0036] The translation vector is obtained based on the position information of the UAV, and the rotation matrix is ​​calculated based on the attitude information of the UAV camera gimbal to obtain the imaging extrinsic parameter matrix.

[0037] Based on the target detection results, the imaging extrinsic matrix, and the camera imaging principle, the transformation from pixel coordinate system to camera coordinate system to world coordinate system is realized to obtain the initial solution of the coordinates of the target to be located in the imaging plane under world coordinates.

[0038] Using bundle adjustment, the three-dimensional coordinates of the target to be located are optimized based on the observation results of each pose of the UAV, so as to obtain the visual target location at the fire scene.

[0039] This invention also discloses an auxiliary firefighting system based on UAV information and image processing technology, the system comprising:

[0040] The image acquisition module is used to acquire images of fire scene using a drone equipped with pre-set equipment;

[0041] The image preprocessing module is used to perform smoke removal preprocessing on the fire scene image to obtain a preprocessed image;

[0042] The target detection module is used to perform target detection on the preprocessed image to obtain target detection results;

[0043] The target localization module is used to perform visual target localization at the fire scene based on the gimbal pose information of the UAV and the target detection results.

[0044] The fire extinguishing assistance module is used to assist in the scheduling of fire-fighting resources based on the target detection results and the visual target positioning, and to complete fire extinguishing assistance based on UAV information and image processing technology.

[0045] Optionally, the process of obtaining the preprocessed image specifically includes:

[0046] A neural network model is constructed based on UNet and ResNet networks. During neural network training, Berlin noise is used to generate model parameters t(x) and c(x). The model parameters t(x) and c(x) are input into an improved atmospheric scattering model to generate smoke images. The neural network model is trained using the smoke images and existing fire scene images to obtain a trained neural network model.

[0047] The fire scene image is input into a trained neural network model to obtain a preprocessed image.

[0048] Optionally, the process of constructing a neural network model based on UNet and ResNet specifically includes:

[0049] The neural network model includes an encoder network and a decoder network;

[0050] The encoder network uses a ResNet network to extract features. The ResNet network contains five neural network layers. The first layer consists of a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer. The input image is resized to a dimension of 640×640×3, and the output is a 320×320×3 feature map. The second, third, and fourth layers are residual network layers, which consist of convolutional layers and residual connections.

[0051] The decoder network consists of five neural network layers. The first layer receives the output of the encoder. The first, second, third, and fourth layers consist of a transposed convolutional layer, a channel concatenation layer, and a residual network layer. The input of each layer first passes through a transposed convolutional layer, then passes through a channel concatenation layer to concatenate the feature map of the same dimension as the feature map in the encoder network, and finally passes through a residual network layer to extract features and output them. The fifth layer consists of two convolutional layers, receiving a 640×640×32 feature map output from the fourth layer of the decoder network. The output dimension is 640×640×3, which is the decoder output.

[0052] Optionally, the process of generating smoke images by using the Berlin noise generation model parameters t(x) and c(x) and inputting the model parameters t(x) and c(x) into the improved atmospheric scattering model specifically includes:

[0053] The generation process of t(x) specifically includes:

[0054] The noise layer number was set to 6, persistence to 0.5, porosity to 2.0, and scale to 250. After normalization, a Burmester noise map P was generated. noise ;

[0055] Based on the aforementioned Berlin noise map, the smoke transmittance map t(x) is generated:

[0056] t(x)=P noise ·(t max -t min )+t min

[0057] Among them, t max t represents the maximum transmittance. min This represents the minimum transmittance.

[0058] The generation process of c(x) specifically includes:

[0059] Based on the aforementioned Berlin noise map, the RGB values ​​c(x) of the smoke and haze are generated:

[0060] c(x)=P noise ·(c max -c min )+c min ;

[0061] Among them, c max c represents the maximum RGB value of smoke and haze. max This represents the minimum RGB value for smoke and haze.

[0062] Substituting the t(x) and c(x) obtained above into the improved atmospheric scattering model, smoke is added to the clear image to obtain a smoke image.

[0063] Optionally, the improved atmospheric scattering model is specifically as follows:

[0064] I(x)=J(x)t(x)+c(x)(1-t(x))

[0065] Where I(x) is the fire scene image; J(x) is the image after ideal defogging processing; t(x) is the smoke transmittance map; and c(x) is the RGB value of the smoke and haze.

[0066] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0067] This invention considers the characteristics of images acquired by UAVs in firefighting scenarios, proposes an image smoke removal preprocessing method, and utilizes an improved YOLOv8 model to perform target detection on the preprocessed image. Based on camera imaging principles, and combining the target detection results with data from the UAV's monocular gimbal sensor, the fire source target is located in three dimensions to assist in the dispatching of firefighting resources. This invention provides richer fire scene information for fire rescue processes at a lower cost, assists in the dispatching of firefighting resources, and improves the intelligence level of firefighting work. Attached Figure Description

[0068] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a flowchart illustrating the firefighting assistance method based on UAV information and image processing technology according to an embodiment of the present invention.

[0070] Figure 2 This is a schematic diagram of the network model of the image dehazing algorithm proposed in an embodiment of the present invention;

[0071] Figure 3 This is a schematic diagram of the improved YOLOv8 network model used in the embodiments of the present invention. Detailed Implementation

[0072] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] Example 1

[0074] A fire-fighting assisted method based on UAV information and image processing technology, such as Figure 1 As shown, the method includes:

[0075] Step S1: Use a drone equipped with preset equipment to collect images of the fire scene.

[0076] Before takeoff, the drone needs to be equipped with a calibrated gimbal camera, various sensors, and wireless communication modules, enabling ground terminal equipment to acquire the drone's 3D position and attitude information from the gimbal camera, as well as the drone's video stream, in real time. The drone then flies over the fire site to monitor the fire.

[0077] Step S2: Perform smoke removal preprocessing on the fire scene image to obtain a preprocessed image.

[0078] like Figure 2 As shown, based on the classic atmospheric scattering model in the field of image dehazing, and with the aim of suppressing smoke in fire scene images, the improved formula is as follows:

[0079] I(x)=J(x)t(x)+c(x)(1-t(x))

[0080] In the formula, I(x) is the image before preprocessing; J(x) is the image after ideal dehazing; t(x) is the smoke transmittance map, which reflects the relative smoke concentration; the higher the smoke concentration, the smaller t(x) is in the region; c(x) is the RGB value of the smoke / haze. t(x) and c(x) are obtained through an encoder neural network and two decoder neural networks. The input image to be processed first passes through the encoder neural network to obtain a feature map with dimensions of 40×40×512, which is used as the input to the decoder network. The two decoders output t(x) and c(x) respectively. Substituting the decoder outputs into the improved haze imaging model formula, we get...

[0081]

[0082] The encoder and decoder neural networks combine UNet and ResNet. The encoder network extracts features based on ResNet and consists of five neural network layers. The first layer consists of convolutional layers, batch normalization layers, ReLU activation function layers, and max pooling layers, resizing the input image to 640×640×3 dimensions and outputting a 320×320×3 feature map. The following four layers are residual network layers, composed of convolutional layers and residual connections, outputting feature maps of 320*320*64, 160*160*128, 80*80*256, and 40*40*512 dimensions respectively. The decoder... The decoder network consists of 5 neural network layers. The first layer receives the output of the encoder. The first four layers have the same structure, consisting of a transposed convolutional layer, a channel concatenation layer, and a residual network layer. The input of each layer first passes through a transposed convolutional layer, then passes through a channel concatenation layer to concatenate the feature map of the same dimension as the encoder network, and finally passes through a residual network layer to extract features and output them. The fifth layer of the decoder network consists of two convolutional layers, receiving a 640×640×32 feature map output from the fourth layer. The output dimension is 640×640×3. The two decoders output t(x) and c(x) respectively.

[0083] During the training process of the neural network, some of the smoke images used were generated based on Berlin noise and an improved atmospheric scattering model. Specifically, the process includes the following steps:

[0084] (1) Generate a Berlin noise map. The number of noise layers (octaves) is set to 6, persistence to 0.5, laminability to 2.0, and scale to 250. The generated Berlin noise map is normalized to obtain P. noise .

[0085] (2) Use Berlin noise to generate the transmittance map t(x). Let the maximum value of the transmittance map t(x) be t. max The minimum value is tm, which is 0.75.in The value is 0.1, according to the formula t(x) = P noise ·(t max -t min )+t min Generate a transmittance map t(x).

[0086] (3) Generate the smoke RGB image c(x) using Berlin noise. Let the maximum value of c(x) be c. max The minimum value is 255, c. min The value is 120, according to the formula c(x) = P noise ·(c max -c min )+c min Generate c(x).

[0087] (4) Substitute the t(x) and c(x) obtained above into the improved atmospheric scattering model to obtain the image after adding haze.

[0088] Step S3: Perform target detection on the preprocessed image to obtain the target detection result.

[0089] like Figure 3 As shown, the object detection adopts an improved YOLOv8 architecture, adding a 160×160 object detection head to enhance object detection capabilities; it changes ordinary upsampling to dynamic upsampling, which is more flexible than nearest neighbor and bilinear interpolation upsampling methods; and it introduces an attention mechanism into the YOLOv8 network structure, adding FasterNetBlock to the shallow layers of the backbone network to increase network depth at a lower cost and extract richer feature information.

[0090] Among them, dysample is a dynamic upsampling mechanism that can dynamically adjust the sampling strategy according to the size and distribution of targets in the input image, thereby adapting to more complex and diverse application environments. Dysample has two working branches. In the first branch, the input feature map is processed by a point sampling generator to obtain a sampling set of sH×sW×2g, which is then input into the bilinear interpolation grid sampling along with the identity mapping from the second branch.

[0091] The coordinate attention mechanism (CA) requires fewer computational resources while preserving location information. This mechanism is applied to YOLOv8. By embedding location information into channel attention, the mobile network can focus on a larger area while avoiding significant computational overhead. Two one-dimensional global pooling operations are used to aggregate the input features in the vertical and horizontal directions into two independent orientation-aware feature maps, respectively. These two feature maps, embedded with specific orientation information, are encoded into two attention maps, each capturing the long-range dependencies of the input feature map along a spatial direction. The two attention maps are then applied to the input feature map via multiplication.

[0092] FasterNetBlock, based on partial convolution Pconv, features a simple structure, few parameters, and fast computation speed. Pconv performs convolution operations on some continuous features in the input channels, while the remaining features are processed through identity mapping, keeping the channels unchanged. This increases the efficiency of floating-point operations by reducing the memory access frequency of floating-point operations.

[0093] Step S4: Based on the gimbal pose information of the UAV and the target detection results, perform visual target localization at the fire scene to complete assisted firefighting based on UAV information and image processing technology. When recording information, considering data processing latency, recording should be done when the UAV is hovering stably, and the recorded information should be different from previously recorded information. A time window queue method is used to identify and record information, as detailed below:

[0094] (1) Create a queue data structure Q that can hold 5 sets of information. When the UAV takes off and starts to perform the mission, enqueue a set of UAV gimbal pose information every 0.3 seconds. When the queue is full, dequeue first and then enqueue.

[0095] (2) Create a time series T. Every 1.5 seconds, calculate the difference between the 5 sets of information in Q. If the difference is greater than a certain threshold, it is determined that the plane is flying and T is assigned the value 1; otherwise, it is determined that the plane is hovering and T is assigned the value 0.

[0096] (3) For time series T, when its value changes from 1 to 0, record the UAV gimbal pose information at time 0.

[0097] The specific methods for visual target localization are as follows:

[0098] (1) Obtain the camera intrinsic parameter matrix according to the camera calibration method. The camera undergoes distortion correction.

[0099] (2) If the origin of the camera coordinate system is set at the optical center of the camera, then points on the imaging plane will have a uniform z-axis. c Its value is the camera focal length f. According to the formula... Mapping the target detection outputs u and v to the camera coordinate system yields X. c =[x c y c z c ] Τ .

[0100] (3) Obtain the rotation matrix based on the UAV camera gimbal attitude information. The UAV camera gimbal attitude can be described by three variables: yaw (relative to true north, positive to the right and negative to the left), roll, and pitch. Based on these three variables, the rotation matrix R = R0 can be obtained. x R y R z ,in In the formula, α is the value of Yaw, β is the value of Roll, and θ is the value of Pitch.

[0101] (4) Obtain the translation vector based on the UAV's position information. Let the origin of the world coordinate system be latitude and longitude lon0, latitude lat0, with the z-axis pointing vertically downwards as positive. Let the position returned by the UAV's sensors be latitude and longitude lon1, latitude lat1, and altitude H. Calculate the latitude and longitude difference between the UAV and the origin and convert it to radians Δlat, Δlon. Then, within a small range (10-50 km), the coordinates of the UAV in the world coordinate system can be obtained, with the x-direction (east-west direction, due east as positive) value being x. u = r × Δlon × cos(lat0), where the value in the y-direction (north-south direction, with true north as positive) is y. u = r × Δlat, where r is the Earth's radius. Further, the translation vector L can be obtained. u =[x u y u -H] Τ .

[0102] (5) Based on the rotation matrix and translation vector, and based on the camera imaging principle, for a set of UAV sensor information, establish the equation R(X) w -L u ) = X c Find X w This allows for the transformation between the pixel coordinate system, the camera coordinate system, and the world coordinate system, resulting in the coordinates of the image of the target to be located in the imaging plane under the world coordinate system.

[0103] (6) In the world coordinate system, let L be the coordinates of the optical center of the UAV camera in the i-th set of UAV sensor information. o =[x 0i y 0i z 0i ] Τ The light center point and X through the cameraw Draw a straight line For each set of UAV sensor information, a straight line can be obtained. Based on the imaging model, the visual 3D localization of the target can be modeled as the following problem: In 3D space, given n straight lines of the above type, find a point such that the sum of the distances from that point to each straight line is minimized. Through analytic geometry and the least squares method, an approximate point can be found that minimizes the difference in distances to each straight line. Let the position of the target point P in the world coordinate system be X = [xyz]. Τ The following least squares optimization problem can be established. This means minimizing the sum of the squared distances from point P to every straight line. According to the formula for the distance from a point to a line in three-dimensional space, where:

[0104]

[0105] Simplifying the above equation, we get:

[0106]

[0107] In the formula, A1, The matrix representing a set of UAV sensor information is calculated using the following formula: The initial coordinates X0 of the target point P in the world coordinate system can be obtained by using the least squares method.

[0108] (7) Nonlinear optimization is performed using bundle adjustment. For each pose observation, according to x(X)=K[R-RL] u X is the pixel coordinate system point x obtained from the world coordinate system point X. For the i-th image, the reprojection error is defined as e i =p i -x(X,R,L u ), where p i =[u,v] T Let be the pixel position of the target in the i-th image. To improve 3D localization accuracy, the reprojection error function needs to be optimized to be as small as possible. That is... By utilizing the gimbal camera pose information acquired by the drone, joint constraints can be added to the optimization objective, at which point the overall optimization function becomes... in λ represents the inverse of the covariance matrix. For UAVs with high positioning accuracy (such as those using RTK technology), λ can be 0. Then, a nonlinear least squares algorithm, such as the Levenberg-Marquardt method, is used for optimization to solve for the position X of the target in the world coordinate system.

[0109] As can be seen from the above mathematical derivation, visual target localization requires information from multiple UAVs in different poses. In order to reduce geometric dilution of precision (GDOP), UAVs should make full use of their maneuverability and form long baselines and large intersection angles in the vertical and horizontal directions from multiple height layers and multiple perspectives during observation, thereby increasing parallax and improving positioning accuracy.

[0110] Considering data processing latency, it is necessary to record data when the drone is hovering and stable. The ground terminal program uses a time window queue method to automatically capture and record the moment when the drone is hovering and stable, as shown below:

[0111] (1) Create a queue data structure Q that can hold 5 sets of information. When the UAV takes off and starts to perform the mission, enqueue a set of UAV gimbal pose information every 0.3 seconds. When the queue is full, dequeue first and then enqueue.

[0112] (2) Create a time series T. Every 1.5 seconds, calculate the difference between the 5 sets of information in Q. If the difference is greater than a certain threshold, it is determined that the plane is flying and T is assigned the value 1; otherwise, it is determined that the plane is hovering and T is assigned the value 0.

[0113] (3) For time series T, when its value changes from 1 to 0, record the UAV gimbal pose information at time 0.

[0114] Step S5: Based on the target detection results and the visual target positioning, assist in scheduling fire-fighting resources to complete fire-fighting assistance based on UAV information and image processing technology.

[0115] This embodiment is applied to assist in the dispatching of fire-fighting resources, and the specific steps are as follows:

[0116] (1) The drone acquires images, which must include fire cannons and fire sources.

[0117] (2) Using the target detection algorithm described in step S3, analyze the image to obtain the attitude (orientation vector) of the ground fire monitor. (Parallel to the ground) Use the visual positioning algorithm described in step S4 to obtain the location of the fire source.

[0118] (3) In the world coordinate system, draw a vector with the position of the fire monitor as the starting point and the position of the fire source as the ending point. vector Decompose into two vectors, one vector A vector perpendicular to the ground (i.e., parallel to the z-axis). Parallel to the ground.

[0119] (4) Find the vector sum vector The included angle is the horizontal angle at which the fire monitor is adjusted. (Vector) The modulus is the vertical distance from the fire monitor to the fire source. Based on this distance and the jet mathematical model, the pitch angle can be adjusted.

[0120] Example 2

[0121] An auxiliary firefighting system based on UAV information and image processing technology, the system includes:

[0122] The image acquisition module is used to acquire images of fire scene using a drone equipped with pre-set equipment;

[0123] The image preprocessing module is used to perform smoke removal preprocessing on the fire scene image to obtain a preprocessed image;

[0124] The target detection module is used to perform target detection on the preprocessed image to obtain target detection results;

[0125] The target localization module is used to perform visual target localization at the fire scene based on the gimbal pose information of the UAV and the target detection results.

[0126] The fire extinguishing assistance module is used to assist in the scheduling of fire-fighting resources based on the target detection results and the visual target positioning, and to complete fire extinguishing assistance based on UAV information and image processing technology.

[0127] The image preprocessing module's workflow specifically includes:

[0128] Smog images are generated based on Berlin noise and an improved atmospheric scattering model;

[0129] A neural network model was constructed based on UNet and ResNet networks;

[0130] The smoke image and existing smoke image data are input into the neural network model to obtain the model parameters;

[0131] The image is dehazed based on model parameters and a classical atmospheric scattering model to obtain a preprocessed image.

[0132] The process of building a neural network model based on UNet and ResNet specifically includes:

[0133] Neural network models include encoder networks and decoder networks;

[0134] The encoder network uses a ResNet network to extract features. The ResNet network contains 5 neural network layers. The first layer consists of a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer. The input image is resized to a dimension of 640×640×3, and the output is a 320×320×3 feature map. The second, third, and fourth layers are residual network layers, which consist of convolutional layers and residual connections.

[0135] The decoder network consists of five neural network layers. The first layer receives the output of the encoder. The first, second, third, and fourth layers consist of transposed convolutional layers, channel concatenation layers, and residual network layers. The input of each layer first passes through a transposed convolutional layer, then passes through a channel concatenation layer to concatenate the feature maps of the same dimension as those in the encoder network, and finally passes through a residual network layer to extract features and output them. The fifth layer consists of two convolutional layers that receive a 640×640×32 feature map output from the fourth layer of the decoder network. The output dimension is 640×640×3, which is the decoder output.

[0136] The workflow of the target localization module specifically includes:

[0137] Based on the camera calibration method, the camera intrinsic parameter matrix is ​​obtained, and distortion processing is performed on the camera based on the camera intrinsic parameter matrix;

[0138] The origin of the orthogonalized camera coordinate system is set as the optical center of the camera to obtain the camera focal length. Based on the focal length and the target detection output, the attitude information of the UAV camera gimbal is obtained.

[0139] Calculate the rotation matrix and translation vector based on the attitude information of the UAV camera gimbal;

[0140] Based on the rotation matrix, the translation vector, and the camera imaging principle, the transformation from pixel coordinate system to camera coordinate system to world coordinate system is realized, and the coordinates of the image of the target to be located in the imaging plane under world coordinates are obtained.

[0141] Bundle adjustment is used to calculate the coordinates of the image of the target to be located in the imaging plane under the world coordinate system to obtain the observation of each pose, and the visual target location at the fire scene is obtained based on the observation results.

[0142] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A fire-fighting assisted method based on UAV information and image processing technology, characterized in that, The method includes: Step S1: Use a drone equipped with preset equipment to collect images of the fire scene; Step S2: Perform smoke removal preprocessing on the fire scene image to obtain a preprocessed image; Step S3: Perform target detection on the preprocessed image to obtain the target detection result; Step S4: Based on the gimbal pose information of the UAV and the target detection results, perform visual target localization at the fire scene; Step S5: Based on the target detection results and the visual target positioning, assist in scheduling fire-fighting resources to complete fire-fighting assistance based on UAV information and image processing technology; In step S2, the process of obtaining the preprocessed image specifically includes: A neural network model is built based on UNet and ResNet networks. During neural network training, Berlin noise is used to generate model parameters. and The model parameters and The improved atmospheric scattering model is input to generate a smoke image. The smoke image and existing fire scene images are used to train the neural network model to obtain a trained neural network model. The fire scene image is input into a trained neural network model to obtain a preprocessed image; The process of constructing a neural network model based on UNet and ResNet specifically includes: The neural network model includes an encoder network and a decoder network; The encoder network uses a ResNet network to extract features. The ResNet network contains five neural network layers. The first layer consists of a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer. The input image is resized to a dimension of 640×640×3, and the output is a 320×320×3 feature map. The second, third, and fourth layers are residual network layers, which consist of convolutional layers and residual connections. The decoder network consists of five neural network layers. The first layer receives the output of the encoder. The first, second, third, and fourth layers consist of a transposed convolutional layer, a channel concatenation layer, and a residual network layer. The input of each layer first passes through a transposed convolutional layer, then passes through a channel concatenation layer to concatenate the feature map of the same dimension as the feature map in the encoder network, and finally passes through a residual network layer to extract features and output them. The fifth layer consists of two convolutional layers, receiving a 640×640×32 feature map output from the fourth layer of the decoder network. The output dimension is 640×640×3, which is the decoder output.

2. The fire-fighting assisted method based on UAV information and image processing technology according to claim 1, characterized in that, The use of Berlin noise generation model parameters and The model parameters and The process of generating a smoke image from an improved atmospheric scattering model specifically includes: The The generation process specifically includes: The noise layer number was set to 6, persistence to 0.5, porosity to 2.0, and scale to 250. After normalization, a Burmester noise plot was generated. ; Based on the Berlin noise map, a smoke transmittance map is generated. : ； in, This represents the maximum transmittance. This represents the minimum transmittance. The The generation process specifically includes: Based on the aforementioned Berlin noise map, RGB values ​​for smoke and haze are generated. : ； in, This represents the maximum RGB value of smoke and haze. This represents the minimum RGB value for smoke and haze. The above results and By substituting the improved atmospheric scattering model into the clear image, smoke is added to obtain a smoke image.

3. The fire-fighting assisted method based on UAV information and image processing technology according to claim 2, characterized in that, The improved atmospheric scattering model is as follows: ； in, Images of fire scene; The image after ideal dehazing; This is a smoke transmittance diagram; These are the RGB values ​​for smoke and haze.

4. The fire-fighting assisted method based on UAV information and image processing technology according to claim 1, characterized in that, In step S4, the process of visual target localization at the fire scene specifically includes: Based on the camera calibration method, the camera intrinsic parameter matrix is ​​obtained, and distortion processing is performed on the camera based on the camera intrinsic parameter matrix; The drone's location information is obtained using the orthodontic drone positioning module; The attitude information of the drone gimbal camera is obtained based on the drone sensors after orthodontic treatment. The translation vector is obtained based on the position information of the UAV, and the rotation matrix is ​​calculated based on the attitude information of the UAV camera gimbal to obtain the imaging extrinsic parameter matrix. Based on the target detection results, the imaging extrinsic matrix, and the camera imaging principle, the transformation from pixel coordinate system to camera coordinate system to world coordinate system is realized to obtain the initial solution of the coordinates of the target to be located in the imaging plane under world coordinates; Using bundle adjustment, the three-dimensional coordinates of the target to be located are optimized based on the observation results of each pose of the UAV, so as to obtain the visual target location at the fire scene.

5. A fire-fighting auxiliary system based on UAV information and image processing technology, the system being used to implement the fire-fighting auxiliary method based on UAV information and image processing technology as described in any one of claims 1-4, characterized in that, The system includes: The image acquisition module is used to acquire images of fire scene using a drone equipped with pre-set equipment; The image preprocessing module is used to perform smoke removal preprocessing on the fire scene image to obtain a preprocessed image; The target detection module is used to perform target detection on the preprocessed image to obtain target detection results; The target localization module is used to perform visual target localization at the fire scene based on the gimbal pose information of the UAV and the target detection results. The fire extinguishing assistance module is used to assist in the scheduling of fire-fighting resources based on the target detection results and the visual target positioning, and to complete fire extinguishing assistance based on UAV information and image processing technology. The process of obtaining a preprocessed image specifically includes: A neural network model is built based on UNet and ResNet networks. During neural network training, Berlin noise is used to generate model parameters. and The model parameters and The improved atmospheric scattering model is input to generate a smoke image. The smoke image and existing fire scene images are used to train the neural network model to obtain a trained neural network model. The fire scene image is input into a trained neural network model to obtain a preprocessed image; The process of constructing a neural network model based on UNet and ResNet specifically includes: The neural network model includes an encoder network and a decoder network; The encoder network uses a ResNet network to extract features. The ResNet network contains five neural network layers. The first layer consists of a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer. The input image is resized to a dimension of 640×640×3, and the output is a 320×320×3 feature map. The second, third, and fourth layers are residual network layers, which consist of convolutional layers and residual connections. The decoder network consists of five neural network layers. The first layer receives the output of the encoder. The first, second, third, and fourth layers consist of a transposed convolutional layer, a channel concatenation layer, and a residual network layer. The input of each layer first passes through a transposed convolutional layer, then passes through a channel concatenation layer to concatenate the feature map of the same dimension as the feature map in the encoder network, and finally passes through a residual network layer to extract features and output them. The fifth layer consists of two convolutional layers, receiving a 640×640×32 feature map output from the fourth layer of the decoder network. The output dimension is 640×640×3, which is the decoder output.

6. The fire-fighting auxiliary system based on UAV information and image processing technology according to claim 5, characterized in that, The use of Berlin noise generation model parameters and The model parameters and The process of generating a smoke image from an improved atmospheric scattering model specifically includes: The The generation process specifically includes: The noise layer number was set to 6, persistence to 0.5, porosity to 2.0, and scale to 250. After normalization, a Burmester noise plot was generated. ; Based on the Berlin noise map, a smoke transmittance map is generated. : ； in, This represents the maximum transmittance. This represents the minimum transmittance. The The generation process specifically includes: Based on the aforementioned Berlin noise map, RGB values ​​for smoke and haze are generated. : ； in, This represents the maximum RGB value of smoke and haze. This represents the minimum RGB value for smoke and haze. The above results and By substituting the improved atmospheric scattering model into the clear image, smoke is added to obtain a smoke image.

Images