Geographic flame segmentation method and apparatus, computer device, and storage medium

Abstract

The present application relates to a geographic flame segmentation method and apparatus, a computer device, and a storage medium. The method comprises: using an unmanned aerial vehicle to acquire a visible light image of a fire region, and using an infrared sensor mounted in the unmanned aerial vehicle to acquire an infrared image of the fire region; fusing the visible light image and the infrared image by means of an image fusion module to generate a multi-source fused image that comprises detailed information and highlights a target; performing feature extraction on the multi-source fused image by means of a ShuffleNet structure-based lightweight feature extraction module to obtain a flame region feature map; and inputting the flame region feature map into a flame segmentation model to obtain a flame segmentation result, and performing georeferencing on the flame segmentation result by means of a geographic coordinate conversion module to output a flame segmentation result having real geographic coordinates. The embodiments of the present application can significantly improve the flame segmentation effect of a model in a fire scenario involving obstructions such as smoke, and greatly reduce the number of model parameters, thereby improving the computational efficiency of the model.

Description

Flame geographic segmentation method, apparatus, computer equipment and storage medium Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and specifically relates to a flame geographic segmentation method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Fire, as a highly destructive natural disaster, not only threatens the safety of the ecological environment but can also cause enormous property damage and casualties. To effectively address this challenge, various remote monitoring and detection technologies have been developed, including ground-based systems, manned aerial vehicle systems, and satellite systems. However, these technologies all face limitations and challenges in practical applications. While ground-based systems are intuitive and easy to operate, their monitoring range is limited and susceptible to terrain and weather conditions. Satellite systems, while capable of covering wider areas, lack sufficient flexibility due to their relatively fixed path planning, making it difficult to accurately pinpoint the exact location and scale of fires in practical applications. Manned aerial vehicle systems, while providing more detailed monitoring data, are significantly limited in practical applications due to their high cost and the high risks posed to operators. Furthermore, while sensor-based fire detection improves the accuracy and efficiency of monitoring to some extent, the potential for false alarms and the high cost of installing multiple sensors over large areas also limit their practical application.

[0003] In recent years, the application of drone technology in fire monitoring has gradually attracted widespread attention. Drones, with their unique advantages such as rapid deployment, high mobility, wide monitoring range, and adjustable viewpoints, have brought new opportunities to fire monitoring. Compared with traditional monitoring technologies, drones can not only capture the specific location and scale of a fire more quickly and accurately, but also conduct more comprehensive and in-depth monitoring and analysis of fires by carrying different sensors and computer equipment. Meanwhile, with the continuous development of modern drone technology, drones have achieved higher levels of intelligence and autonomy. Some advanced drones can autonomously complete complex operations such as flight missions, obstacle avoidance, and recharging, greatly improving their safety and reliability in practical applications. Furthermore, modern drones can be equipped with high-performance computing devices such as micro edge TPU / GPU platforms to achieve real-time data processing and analysis, further improving the accuracy and efficiency of fire monitoring.

[0004] In the field of computer vision, various algorithms have emerged for fire detection, used to identify fire phenomena in video or image sequences. For example, Namozov et al. proposed an innovative deep convolutional neural network algorithm aimed at achieving high-precision fire and smoke image detection. Meanwhile, Zhang et al. proposed a deep learning method specifically for forest fire detection. This method trains full-image and fine-grained patch fire classifiers in a connected deep convolutional neural network, running fire detection in a cascaded manner. Although these algorithms can effectively determine whether a fire exists in an image, they typically only provide information on the presence or absence of a fire, and cannot achieve precise segmentation of flames, i.e., cannot specifically identify which pixels in the image belong to flames.

[0005] Flame segmentation, as the cornerstone of fire detection and modeling, is undeniably crucial. Accurate segmentation of fire areas in images allows for the acquisition of critical information such as the fire's location, area, and spread rate, providing strong support for subsequent emergency response and decision-making. In recent years, numerous researchers have made significant progress in this field. For example, Akhloufi et al. proposed a deep convolutional neural network (DCNN) called Deep-Fire Network. This network, based on the U-Net architecture, uses visible light RGB images of forest fires as input, processes them through a model, and outputs a binary mask to identify flame pixels in the image. While existing research has achieved good results in flame segmentation accuracy, many problems still exist in practical applications. First, existing flame segmentation algorithms typically use datasets acquired from oblique photography angles, but the segmented flame areas obtained from this perspective do not represent the true extent of the flame. Second, when performing flame segmentation, existing algorithms are usually limited to using various visible light data, ignoring the role of infrared data, making it difficult to effectively detect and segment flame areas when there is smoke or other obstructions. In addition, the model size of various flame segmentation algorithms also has a significant impact on their practical applications. The large number of model parameters and computational performance requirements make it difficult to deploy on edge computing devices, which is not conducive to real-time monitoring of hazard sources and data acquisition.

[0006] Summary of the Invention

[0007] This application provides a flame geographic segmentation method, apparatus, computer device, and storage medium, which aims to at least partially solve one of the aforementioned technical problems in the prior art.

[0008] To address the above problems, this application provides the following technical solution:

[0009] A flame geographic segmentation method, comprising:

[0010] The drone was used to acquire visible light images of the fire area, and the infrared sensor on the drone was used to acquire infrared images of the fire area.

[0011] The visible light image and infrared image are fused by the image fusion module to generate a multi-source fused image that includes detailed information and highlights the target;

[0012] The feature map of the flame region is obtained by extracting features from the multi-source fused image using a lightweight feature extraction module based on the ShuffleNet structure.

[0013] The flame region feature map is input into the flame segmentation model to obtain the flame segmentation result. The flame segmentation result is then geo-registered using a geographic coordinate transformation module to output a flame segmentation result with real geographic coordinates.

[0014] The technical solution adopted in this application embodiment further includes: after acquiring visible light images of the fire area using a drone and acquiring infrared images of the fire area using an infrared sensor mounted on the drone, it further includes:

[0015] The visible light and infrared images are preprocessed using image adjustment operations, and then the preprocessed visible light and infrared images are subjected to random rotation, random flipping, and random scaling transformation data enhancement processing to obtain data-enhanced visible light and infrared images.

[0016] The technical solution adopted in this application embodiment further includes: fusing the visible light image and the infrared image through an image fusion module to generate a multi-source fused image that includes detailed information and highlights the target, specifically:

[0017] The image fusion module comprises two parts: a feature extractor and an image reconstructor. The feature extractor includes an infrared feature extraction branch and a visible light feature extraction branch. Each feature extraction branch includes a common convolutional layer and two dense gradient residual modules. The common convolutional layer uses ordinary convolutional layers to extract shallow features that can distinguish between the light image and the infrared image. The two dense gradient residual modules extract fine-grained features that can distinguish between the light image and the infrared image from the shallow features. After fusing the fine-grained features of the visible light image and the infrared image using a stitching strategy, the images are fed into the image reconstructor for feature aggregation and image reconstruction to generate a multi-source fused image.

[0018] The technical solution adopted in this application embodiment also includes: the two dense gradient residual modules each include a main branch and a residual branch parallel to the main branch. The main branch combines two 3×3 convolutional layers and one 1×1 convolutional layer to capture multi-scale feature information and introduces dense connections so that each convolutional layer uses the features extracted by the previous layer. The residual branch uses gradient operation to measure the gradient magnitude of the features and uses a 1×1 convolutional layer to adjust the channel dimension. Finally, the outputs of the main branch and the residual branch are fused by element-wise addition to achieve the integration of deep features and fine-grained features.

[0019] The technical solution adopted in this application embodiment also includes: extracting features from the multi-source fusion image using a lightweight feature extraction module based on the ShuffleNet structure to obtain a flame region feature map, specifically:

[0020] The ShuffleNet structure uses grouped convolution and channel reassembly operations for feature extraction. The grouped convolution divides the input feature map into groups, and then performs convolution on each group separately. The channel reassembly randomly shuffles and recombines the information between different groups obtained from the first layer of grouped convolution, and uses it as the input for the next layer of grouped convolution. This allows different groups in the next layer of grouped convolution to obtain information and features from all groups in the previous layer of grouped convolution, enabling information to flow between different groups.

[0021] The technical solution adopted in this application embodiment further includes: inputting the flame region feature map into the flame segmentation model to obtain the flame segmentation result, and performing geographic registration on the flame segmentation result through the geographic coordinate transformation module to output the flame segmentation result with real geographic coordinates, specifically as follows:

[0022] Let the coordinates of image point a in the two-dimensional image plane coordinate system be (x, y). With the photography center S as the origin, and the x'y' axes parallel to the x and y axes of the image plane coordinate system, and the z' axis coinciding with the optical axis, establish an image space coordinate system. Then the coordinates of image point a in the image space coordinate system are (x, y, -f), where f represents the focal length. With the photography center S as the origin, establish an auxiliary image space coordinate system parallel to the ground measurement coordinate system. Let the coordinates of image point a in the auxiliary image space coordinate system be (u, v, w). Then the coordinate transformation relationship for image point a from image space coordinates (x, y, -f) to auxiliary image space coordinates (u, v, w) is:

[0023] Where R is the rotation matrix, and is determined by the direction cosine a i b i c i Composed of (i = 1, 2, 3), a i b i ci (i = 1, 2, 3) are derived from the elevation angle in the image exterior orientation elements. The sine and cosine of the roll angle ω and the yaw angle k are calculated using the following formulas:

[0024] Let the coordinates of image point a corresponding to ground point A in the ground measurement coordinate system be (X... A ,Y A Z A The coordinates of the photography center S are (X... S ,Y S Z S According to the principle of similar triangles, the auxiliary coordinates (u, v, w) of image point a in image space are similar to the ground measurement coordinates (X, v, w) of the corresponding ground point A. A ,Y A Z A The conversion relationship between them is:

[0025] Where λ is the scaling factor, satisfying

[0026] Combining the above formulas, we can obtain:

[0027] Assuming the camera focal length f and the image exterior orientation element X are known... S Y S Z S Pitch angle The roll angle ω and yaw angle k are used to calculate the pixel coordinates xy of the ground point in the UAV image by solving the collinearity equation. Given the length and width of the UAV image, the spatial coordinates of the ground point corresponding to the four corner points of the UAV image are calculated using the collinearity equation to determine the image coverage area. The true geographic coordinates of the flame segmentation result are then obtained using the image coverage area and the pixel coordinates xy.

[0028] The technical solution adopted in this application embodiment further includes: after inputting the flame region feature map into the flame segmentation model to obtain the flame segmentation result, and performing geographic registration on the flame segmentation result through the geographic coordinate transformation module to output the flame segmentation result with real geographic coordinates, it further includes:

[0029] Based on the flame segmentation results, the flame segmentation model is optimized using cross-entropy loss as the loss function, and tested based on pixel accuracy, average intersection-over-union (IoU), and weighted IoU; where pixel accuracy is calculated as the total number of correctly classified pixels, and the calculation formula is:

[0030] The average crossover ratio (CVR) estimates the CVR value for each category and approximates the average CVR for all categories:

[0031] The weighted intersection-union ratio is weighted according to the frequency of each object category:

[0032] In the above formula, n ii t represents the correctly classified pixels in the flame segmentation result. i Let n be the total number of pixels of class i. ij n represents the number of pixels that belong to class i but are predicted to be class j. cl The total number of categories is represented by PA, the pixel accuracy is represented by MIoU, the average crossover ratio is represented by FWIoU, and the weighted crossover ratio is represented by FWIoU.

[0033] Another technical solution adopted in this application embodiment is: a flame geographic segmentation device, comprising:

[0034] Image acquisition module: used to acquire visible light images of the fire area using the drone, and to acquire infrared images of the fire area using the infrared sensor on the drone;

[0035] Image fusion module: used to fuse the visible light image and infrared image to generate a multi-source fused image that includes detailed information and highlights the target;

[0036] Lightweight feature extraction module: used to extract features from the multi-source fused image based on the ShuffleNet structure to obtain a flame region feature map;

[0037] Geographic coordinate transformation module: This module is used to input the flame region feature map into the flame segmentation model, obtain the flame segmentation result, perform geographic registration on the flame segmentation result, and output the flame segmentation result with real geographic coordinates.

[0038] Another technical solution adopted in this application embodiment is: a computer device, the computer device including a processor and a memory coupled to the processor, wherein,

[0039] The memory stores program instructions for implementing the flame geographic segmentation method;

[0040] The processor is used to execute the program instructions stored in the memory to control the flame geographic segmentation method.

[0041] Another technical solution adopted in this application embodiment is: a storage medium storing processor-executable program instructions, the program instructions being used to execute the flame geographic segmentation method.

[0042] Compared to existing technologies, the beneficial effects of the embodiments of this application are as follows: The flame geographic segmentation method, apparatus, computer equipment, and storage medium of the embodiments of this application introduce UAV multi-source video fusion technology to fuse visible light images and infrared images extracted by UAVs. By supplementing and improving the visible light images with infrared images, richer information is provided, and the accuracy and robustness of the algorithm in identifying flames are enhanced. A lightweight feature extraction module is designed using the ideas of grouped convolution and channel reorganization, combined with advanced deep learning algorithms to achieve accurate flame segmentation, significantly reducing the number of model parameters, improving the model's computational speed, and maintaining consistent segmentation results. The embodiments of this application can significantly improve the flame segmentation effect of the model in fire scenarios with smoke obscuring the view, and greatly reduce the number of model parameters, improving the model's computational efficiency. This facilitates the deployment of algorithms on edge computing devices such as UAVs for real-time flame geographic segmentation in fire emergency scenarios. Attached Figure Description

[0043] Figure 1 is a flowchart of the flame geographic segmentation method according to an embodiment of this application;

[0044] Figure 2 is a schematic diagram of the network architecture of the image fusion module in an embodiment of this application;

[0045] Figure 3 is a schematic diagram of the GRDB module structure according to an embodiment of this application;

[0046] Figure 4 is a schematic diagram of the ShuffleNet structure according to an embodiment of this application, wherein (a) is the traditional ShuffleNet structure design, (b) is the improved ShuffleNet structure, and (c) is the ShuffleNet downsampling module;

[0047] Figure 5 is a schematic diagram of channel recombination of two stacked grouped convolutions;

[0048] Figure 6 is a schematic diagram of the UAV imaging model;

[0049] Figure 7 is a schematic diagram of the structure of the flame geographic segmentation device according to an embodiment of this application;

[0050] Figure 8 is a schematic diagram of the computer device structure according to an embodiment of this application;

[0051] Figure 9 is a schematic diagram of the structure of the storage medium according to an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] The terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or computer device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or computer devices.

[0054] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] Specifically, please refer to Figure 1, which is a flowchart of the flame geographic segmentation method according to an embodiment of this application. The flame geographic segmentation method according to an embodiment of this application includes the following steps:

[0056] S100: Uses a drone to acquire visible light images of the fire area, and at the same time uses the infrared sensor on the drone to acquire infrared images of the fire area.

[0057] In this step, drones can safely and quickly deploy and monitor fire areas. Furthermore, the visible light images acquired from the drone's orthogonal viewpoint can effectively avoid the influence of the outer flame of the fire, which is beneficial for obtaining the true fire area. At the same time, drones carrying infrared sensors have gradually become standard equipment. Using infrared sensors, it is easy to obtain infrared images of the fire area. There is a significant temperature difference between the fire area and the background in the infrared image, which makes the fire area more prominent.

[0058] S110: Perform preprocessing and data augmentation on visible light and infrared images to obtain data-enhanced visible light and infrared images;

[0059] In this step, since visible light images are usually acquired by various cameras and have high resolution, directly feeding high-resolution images into a deep learning network would significantly increase the computational complexity of the network, making the training process exceptionally time-consuming, even exceeding the capabilities of the hardware computing equipment. Therefore, this embodiment first uses image resizing operations to preprocess the visible light and infrared images to reduce their resolution. Then, it performs data augmentation processing on the preprocessed visible light and infrared images, such as random rotation, random flipping, and random scaling transformations, to improve model performance, prevent overfitting of the flame segmentation model, increase the flame segmentation model's ability to recognize targets at different angles and scales, and make the flame segmentation model more easily generalized to different scenes and conditions.

[0060] S120: The image fusion module fuses the preprocessed visible light image and infrared image to generate a multi-source fused image that includes detailed information and highlights the target;

[0061] In this step, to enable the flame segmentation model to combine the characteristics of visible light and infrared images, an image fusion module is first introduced to fuse the light and infrared images. Specifically, the network architecture of the image fusion module is shown in Figure 2, which includes a feature extractor and an image reconstructor. The feature extractor includes two parallel feature extraction branches: an infrared feature extraction branch and a visible light feature extraction branch. Each feature extraction branch includes a common convolutional layer and two dense gradient residual blocks (GRDBs). The common convolutional layer uses a 3×3 kernel and a Leaky Rectified Linear Unit (LReLU) activation function to extract shallow features from the light and infrared images. Then, the two GRDB modules further extract fine-grained features from the shallow features.

[0062] Further, please refer to Figure 3, which is a schematic diagram of the GRDB module structure according to an embodiment of this application. The GRDB module includes a main branch and a residual branch running parallel to the main branch. The main branch combines two 3×3 convolutional layers and one 1×1 convolutional layer to capture multi-scale feature information. It also introduces the concept of dense connections, enabling each convolutional layer to effectively utilize the features extracted by the previous layer, thereby improving the feature extraction capability of the entire network. The residual branch uses gradient operations to measure the gradient magnitude of features. To maintain compatibility between features in different convolutional layers, the residual branch uses a 1×1 convolutional layer to adjust the channel dimension, helping the network capture subtle changes between features. Then, the outputs of the main branch and the residual branch are fused using element-wise addition to achieve the integration of deep features and fine-grained features.

[0063] Finally, a stitching strategy is employed to fuse the fine-grained features of the visible light and infrared images. The fusion result is then fed into an image reconstructor for feature aggregation and image reconstruction, generating a multi-source fused image. The image reconstructor consists of three 3×3 convolutional layers and one 1×1 convolutional layer. All 3×3 convolutional layers use LReLU as the activation function, while the 1×1 convolutional layer uses Tanh as the activation function.

[0064] S130: Feature extraction is performed on the multi-source fusion image using a lightweight feature extraction module based on the ShuffleNet structure to obtain the flame region feature map;

[0065] In this step, to address the problem that flame segmentation models are difficult to deploy and perform real-time segmentation in practical applications, this application embodiment introduces a lightweight feature extraction module to improve the existing flame segmentation model. While ensuring the segmentation effect, it reduces the number of model parameters and improves computational efficiency, laying the foundation for real-time flame geographic segmentation at the edge.

[0066] Specifically, to ensure the performance of the flame segmentation model while reducing computational load and model size, the lightweight feature extraction module is based on the ShuffleNet structure, as shown in Figure 4. The ShuffleNet structure uses grouped convolutions and channel reorganization operations for feature extraction, reducing the computational complexity of dense 1×1 convolutions used in traditional flame segmentation models. Specifically, the traditional ShuffleNet structure design is shown in Figure 4(a), where the 3×3 convolution is replaced with a 3×3 depthwise convolution (i.e., DWConv in Figure 4(a)). 1×1 convolutions are used before and after the convolutions for dimensionality reduction and expansion, including BN and ReLU layers, and finally, they are summed using Add. This embodiment improves upon Figure 4(a) by replacing the first 1×1 convolution used to reduce the number of channels with a 1×1 grouped convolution plus channel reorganization, and replacing the second 1×1 convolution used to expand the number of channels with a 1×1 grouped convolution, thus forming a ShuffleNet module with the same input and output feature map size, as shown in Figure 4(b). Figure 4(c) shows the ShuffleNet downsampling module. To halve the size of the output feature map, the stride of the 3×3 depthwise convolution in the ShuffleNet module is set to 2, and the ReLU activation function after the depthwise convolution is removed. At the same time, average pooling is used on the short-circuit path, and the original element-wise summation is replaced by channel concatenation (Concat) operation. This turns the original numerical addition operation into dimensional addition. Higher dimensions can give the network more channels, thereby improving the network performance.

[0067] Furthermore, grouped convolution can help the flame segmentation model distribute the model across two GPUs for training. Grouped convolution groups the input feature map and then performs convolution on each group separately. Compared to traditional full convolution, grouped convolution can reduce the number of model parameters to 1 / g, where g represents the number of groups. However, due to the fixed groups, the information between the groups cannot be fused, thus weakening the feature extraction ability of each convolution kernel and affecting the performance of the entire network. Therefore, this embodiment utilizes channel recombination to randomly shuffle and recombine the information between different groups obtained from the first layer of grouped convolution as the input to the next layer of grouped convolution. This allows different groups in the next layer of grouped convolution to obtain information and features from all groups in the previous layer of grouped convolution. The input and output channels are fully correlated, and the features of each group convolution are no longer limited to a single group but come from all groups, thereby achieving the purpose of information flow between different groups. Specifically, Figure 5 shows a schematic diagram of channel recombination for stacking two grouped convolutions.

[0068] S140: Input the flame region feature map into the flame segmentation model to obtain the flame segmentation result, and perform georegistration on the flame segmentation result through the geographic coordinate transformation module to output the flame segmentation result with real geographic coordinates;

[0069] In this step, the flame segmentation results are geo-registered by solving the spatial collinearity equation, utilizing the UAV's attitude information and accurate geographic location information in RTK mode. Establishing the spatial mapping relationship between image points and ground points is crucial in image geo-registration. At the moment of aerial image acquisition, since the image point, the camera center, and the corresponding ground point are coplanar, they satisfy the spatial collinearity condition. Therefore, the spatial mapping relationship between the image point and the corresponding ground point can be constructed using the collinearity condition equation. Figure 6 shows a schematic diagram of the UAV imaging model. Let the coordinates of image point a in the two-dimensional image plane coordinate system be (x, y). With the photography center S as the origin, the x'y' axes are parallel to the xy axes of the image plane coordinate system, and the z' axis coincides with the optical axis to establish an image space coordinate system. Then the coordinates of image point a in the image space coordinate system are (x, y, -f), where f represents the focal length. With the photography center S as the origin, establish an auxiliary image space coordinate system parallel to the ground measurement coordinate system. Let the coordinates of image point a in the auxiliary image space coordinate system be (u, v, w). Then the coordinate transformation relationship of image point a from image space coordinates (x, y, -f) to image space auxiliary coordinates (u, v, w) is expressed by equation (1):

[0070] Where R is the rotation matrix, and is determined by the direction cosine a i b i c i Composed of (i = 1, 2, 3), a i b i c i (i = 1, 2, 3) are derived from the elevation angle in the image exterior orientation elements. The sine and cosine of the roll angle ω and the yaw angle k are calculated using the following formulas:

[0071] The ground survey coordinate system is a left-handed rectangular coordinate system in space. Its XY plane is the same as the plane rectangular coordinate system of the Gauss-Kruger projection in geodesy, and its elevation is based on the Yellow Sea elevation system of my country. Let the coordinates of image point a corresponding to ground point A in the ground survey coordinate system be (X... A ,Y A Z A The coordinates of the photography center S are (X... S ,Y S Z S According to the principle of similar triangles, the auxiliary coordinates (u, v, w) of image point a in image space are similar to the ground measurement coordinates (X, v, w) of the corresponding ground point A.A ,Y A Z A There exists a transformation relationship as shown in formula (3) between them:

[0072] Where λ is the scaling factor, satisfying

[0073] Combining formulas (1) and (3), we can obtain:

[0074] From the above derivation process, it can be seen that, assuming the camera focal length f and the image exterior orientation element X are known... S Y S Z S Pitch angle The roll angle ω and yaw angle k are used to solve the collinearity equation to obtain the pixel coordinates xy of the ground point in the UAV image. Given the length and width of the UAV image, the collinearity equation can be used to calculate the spatial coordinates of the ground point corresponding to the four corner points of the UAV image, thus initially determining the image coverage area. The actual geographic coordinates of the flame segmentation result can be obtained using the image coverage area and the pixel coordinates xy.

[0075] S150: Based on the flame segmentation results, the flame segmentation model is optimized using cross-entropy loss as the loss function, and the flame segmentation model is tested based on pixel accuracy, average cross-union ratio, and weighted cross-union ratio.

[0076] In this step, the dataset is trained for 100 rounds with a batch size of 2 and an initial learning rate of 0.05. A poly strategy is used to automatically adjust the learning rate during training. Cross-entropy loss is used as the loss function to supervise the training process, and the model is tested based on pixel accuracy, average cross-union ratio (CUI), and weighted CUI. The model with the best results is saved. Pixel accuracy is calculated as the total number of correctly classified pixels, as shown in formula (5):

[0077] The average crossover ratio (CWR) is first estimated for each category, and then the average CWR for all categories is approximately calculated, as shown in formula (6):

[0078] The weighted crossover ratio (CUP) is an extended version of the average CUP, where the CUP is weighted according to the frequency of each object class, as shown in formula (7):

[0079] In formulas (5) to (7), n ii t represents the correctly classified pixels in the flame segmentation result. i Let n be the total number of pixels of class i. ijn represents the number of pixels that belong to class i but are predicted to be class j. cl This indicates the total number of categories. PA represents pixel accuracy, MIoU represents average intersection-over-union ratio, and FWIoU represents weighted intersection-over-union ratio.

[0080] Based on the above, the flame geographic segmentation method in this application introduces UAV multi-source video fusion technology to fuse visible light and infrared images extracted by the UAV. By supplementing and improving the visible light image with infrared imagery, it not only provides richer information but also enhances the accuracy and robustness of the algorithm in identifying flames. A lightweight feature extraction module is designed using grouped convolution and channel reorganization, combined with advanced deep learning algorithms to achieve accurate flame segmentation, significantly reducing the number of model parameters, improving computational speed, and maintaining consistent segmentation results. This application embodiment can significantly improve the flame segmentation effect in fire scenarios with smoke obscuring the image, greatly reduce the number of model parameters, and improve computational efficiency. It facilitates the deployment of algorithms on edge computing devices such as UAVs for real-time flame geographic segmentation in fire emergency scenarios.

[0081] Please refer to Figure 7, which is a schematic diagram of the structure of the flame geographic segmentation device according to an embodiment of this application. The flame geographic segmentation device 40 according to an embodiment of this application includes:

[0082] Image acquisition module 41: used to acquire visible light images of the fire area using a drone, and to acquire infrared images of the fire area using an infrared sensor mounted on the drone;

[0083] Image fusion module 42: used to fuse the visible light image and the infrared image to generate a multi-source fused image that includes detailed information and highlights the target;

[0084] Lightweight feature extraction module 43: used to extract features from the multi-source fused image based on the ShuffleNet structure to obtain a flame region feature map;

[0085] Geographic coordinate transformation module 44: is used to input the flame region feature map into the flame segmentation model, obtain the flame segmentation result, perform geographic registration on the flame segmentation result, and output the flame segmentation result with real geographic coordinates.

[0086] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0087] The apparatus provided in this application can be applied to the foregoing method embodiments. For details, please refer to the description of the above method embodiments, which will not be repeated here.

[0088] Please refer to Figure 8, which is a schematic diagram of the structure of a computer device according to an embodiment of this application. The computer device 50 includes:

[0089] Memory 51 storing executable program instructions;

[0090] Processor 52 connected to memory 51;

[0091] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: acquiring visible light images of the fire area using a drone, and acquiring infrared images of the fire area using an infrared sensor mounted on the drone; fusing the visible light images and infrared images using an image fusion module to generate a multi-source fused image that includes detailed information and highlights the target; extracting features from the multi-source fused image using a lightweight feature extraction module based on the ShuffleNet structure to obtain a flame area feature map; inputting the flame area feature map into a flame segmentation model to obtain a flame segmentation result, and performing georegistration on the flame segmentation result using a geographic coordinate transformation module to output a flame segmentation result with real geographic coordinates.

[0092] The processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0093] Please refer to Figure 9, which is a schematic diagram of the structure of the storage medium in an embodiment of this application. The storage medium in this embodiment stores program instructions 61 capable of implementing the following steps: acquiring a visible light image of the fire area using a drone, and acquiring an infrared image of the fire area using an infrared sensor mounted on the drone; fusing the visible light image and the infrared image using an image fusion module to generate a multi-source fused image including detailed information and highlighting the target; extracting features from the multi-source fused image using a lightweight feature extraction module based on the ShuffleNet structure to obtain a flame area feature map; inputting the flame area feature map into a flame segmentation model to obtain a flame segmentation result, and performing georegistration on the flame segmentation result using a geographic coordinate transformation module to output a flame segmentation result with real geographic coordinates. The program instructions 61 can be stored in the aforementioned storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network computer device, etc.) or processor to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program instructions, or terminal computer devices such as computers, servers, mobile phones, and tablets. Servers can be standalone servers or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for segmenting flame geographic locations, characterized in that, include: The drone was used to acquire visible light images of the fire area, and the infrared sensor on the drone was used to acquire infrared images of the fire area. The visible light image and infrared image are fused by the image fusion module to generate a multi-source fused image that includes detailed information and highlights the target; Feature extraction of the multi-source fused image is performed using a lightweight feature extraction module based on the ShuffleNet structure to obtain a flame region feature map; The flame region feature map is input into the flame segmentation model to obtain the flame segmentation result. The flame segmentation result is then geo-registered using the geographic coordinate transformation module to output the flame segmentation result with real geographic coordinates.

2. The flame geographic segmentation method according to claim 1, characterized in that, After acquiring visible light images of the fire area using a drone and acquiring infrared images of the fire area using an infrared sensor mounted on the drone, the method further includes: The visible light and infrared images are preprocessed using image adjustment operations, and then the preprocessed visible light and infrared images are subjected to random rotation, random flipping, and random scaling transformation data enhancement processing to obtain data-enhanced visible light and infrared images.

3. The flame geographic segmentation method according to claim 2, characterized in that, The process of fusing the visible light image and the infrared image through an image fusion module to generate a multi-source fused image that includes detailed information and highlights the target is as follows: The image fusion module comprises two parts: a feature extractor and an image reconstructor. The feature extractor includes an infrared feature extraction branch and a visible light feature extraction branch, each of which includes common features. The system consists of a convolutional layer and two dense gradient residual modules. The common convolutional layer uses ordinary convolutional layers to extract shallow features from the visible light image and the infrared image, respectively. The two dense gradient residual modules extract fine-grained features from the shallow features to the visible light image and the infrared image. After fusing the fine-grained features of the visible light image and the infrared image using a stitching strategy, the images are fed into the image reconstructor for feature aggregation and image reconstruction to generate a multi-source fused image.

4. The flame geographic segmentation method according to claim 3, characterized in that, The two dense gradient residual modules each include a main branch and a residual branch running parallel to the main branch. The main branch combines two 3×3 convolutional layers and one 1×1 convolutional layer to capture multi-scale feature information and introduces dense connections so that each convolutional layer utilizes the features extracted by the previous layer. The residual branch uses gradient operations to measure the gradient magnitude of the features and uses a 1×1 convolutional layer to adjust the channel dimension. Finally, the outputs of the main branch and the residual branch are fused using element-wise addition to achieve the integration of deep features and fine-grained features.

5. The flame geographic segmentation method according to any one of claims 1 to 4, characterized in that, The step of extracting features from the multi-source fused image using a lightweight feature extraction module based on the ShuffleNet structure to obtain a flame region feature map is as follows: The ShuffleNet structure uses grouped convolution and channel reassembly operations for feature extraction. The grouped convolution divides the input feature map into groups, and then performs convolution on each group separately. The channel reassembly randomly shuffles and recombines the information between different groups obtained from the first layer of grouped convolution, and uses it as the input for the next layer of grouped convolution. This allows different groups in the next layer of grouped convolution to obtain information and features from all groups in the previous layer of grouped convolution, enabling information to flow between different groups.

6. The flame geographic segmentation method according to claim 5, characterized in that, The process involves inputting the flame region feature map into the flame segmentation model to obtain the flame segmentation result, and then performing georegistration on the flame segmentation result using a geographic coordinate transformation module to output a flame segmentation result with real geographic coordinates. Specifically: Let the coordinates of image point a in the two-dimensional image plane coordinate system be (x, y). With the photography center S as the origin, and the x'y' axes parallel to the x and y axes of the image plane coordinate system, and the z' axis coinciding with the optical axis, establish an image space coordinate system. Then the coordinates of image point a in the image space coordinate system are (x, y, -f), where f represents the focal length. With the photography center S as the origin, establish an auxiliary image space coordinate system parallel to the ground measurement coordinate system. Let the coordinates of image point a in the auxiliary image space coordinate system be (u, v, w). Then the coordinate transformation relationship for image point a from image space coordinates (x, y, -f) to auxiliary image space coordinates (u, v, w) is: Where R is the rotation matrix, and is determined by the direction cosine a i b i c i Composed of (i = 1, 2, 3), a i b i c i (i = 1, 2, 3) are derived from the elevation angle in the image exterior orientation elements. The sine and cosine of the roll angle ω and the yaw angle k are calculated using the following formulas: Let the coordinates of image point a corresponding to ground point A in the ground measurement coordinate system be (X... A ,Y A Z A The coordinates of the photography center S are (X... S ,Y S Z S According to the principle of similar triangles, the auxiliary coordinates (u, v, w) of image point a in image space are similar to the ground measurement coordinates (X, v, w) of the corresponding ground point A. A ,Y A Z A The conversion relationship between them is: Where λ is the scaling factor, satisfying Combining the above formulas, we can obtain: Assuming the camera focal length f and the image exterior orientation element X are known... S Y S Z S Pitch angle The roll angle ω and yaw angle k are used to calculate the pixel coordinates xy of the ground point in the UAV image by solving the collinearity equation. Given the length and width of the UAV image, the spatial coordinates of the ground point corresponding to the four corner points of the UAV image are calculated using the collinearity equation to determine the image coverage area. The true geographic coordinates of the flame segmentation result are then obtained using the image coverage area and the pixel coordinates xy.

7. The flame geographic segmentation method according to claim 6, characterized in that, After inputting the flame region feature map into the flame segmentation model to obtain the flame segmentation result, and performing georegistration on the flame segmentation result through the geographic coordinate transformation module to output the flame segmentation result with real geographic coordinates, the process further includes: Based on the flame segmentation results, the flame segmentation model is optimized using cross-entropy loss as the loss function, and tested based on pixel accuracy, average intersection-over-union (IoU), and weighted IoU; where pixel accuracy is calculated as the total number of correctly classified pixels, and the calculation formula is: The average crossover ratio (CVR) estimates the CVR value for each category and approximates the average CVR for all categories: The weighted intersection-union ratio is weighted according to the frequency of each object category: In the above formula, n ii t represents the correctly classified pixels in the flame segmentation result. i n is the total number of pixels of class i. ij n represents the number of pixels that belong to class i but are predicted to be class j. cl The total number of categories is represented by PA, the pixel accuracy is represented by MIoU, the average crossover ratio is represented by FWIoU, and the weighted crossover ratio is represented by FWIoU.

8. A flame-based geographic segmentation device, characterized in that, include: Image acquisition module: used to acquire visible light images of the fire area using the drone, and to acquire infrared images of the fire area using the infrared sensor on the drone; Image fusion module: used to fuse the visible light image and infrared image to generate a multi-source fused image that includes detailed information and highlights the target; Lightweight feature extraction module: used to extract features from the multi-source fused image based on the ShuffleNet structure to obtain a flame region feature map; Geographic coordinate transformation module: This module is used to input the flame region feature map into the flame segmentation model, obtain the flame segmentation result, perform geographic registration on the flame segmentation result, and output the flame segmentation result with real geographic coordinates.

9. A computer device, characterized in that, The computer device includes a processor and a memory coupled to the processor, wherein, The memory stores program instructions for implementing the flame geographic segmentation method according to any one of claims 1-7; The processor is used to execute the program instructions stored in the memory to control the flame geographic segmentation method.

10. A storage medium, characterized in that, The device stores processor-executable program instructions for performing the flame geographic segmentation method according to any one of claims 1 to 7.

Images