Semantic segmentation method, system and equipment for aerial image and medium
By adopting a dual-stream convolutional neural network model based on attention mechanism in drone aerial image processing, fusing visible light and thermal infrared image features, the problem of taking into account both accuracy and efficiency in the existing technology is solved, and the accuracy and stability of semantic segmentation are improved .
Patent Information
- Application Number
- CN202410962609.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art is difficult to take into account both semantic segmentation accuracy and efficiency when fusing visible light images and thermal infrared images, resulting in a degradation of performance in environments with dim light or low visibility.
Using a dual-stream convolutional neural network model based on attention mechanism, feature extraction and complementary perception fusion of RGB-T images taken by drone aerial shots, capture the weight of different mode data through the channel attention module and the spatial attention module, and improve the weight of data. Spatial resolution and positioning accuracy.
The feature utilization rate of the RGB-T image semantic segmentation model and the semantic segmentation accuracy under different lighting conditions are improved, and the generalization ability of the model and the segmentation ability of complex scenes are enhanced.
Smart Images

Figure CN119027834A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a method, system, device and medium for semantic segmentation of aerial images. Background Art
[0002] The purpose of semantic segmentation of drone images is to analyze the content of aerial images collected by drones at the pixel level, so as to obtain results with pixel-level semantic category annotations. This technology has a wide range of applications in many fields such as urban planning, smart cities, medical and health care, and geographic information system construction.
[0003] With the development of technologies such as autonomous driving, scene understanding, and virtual reality, image semantic segmentation has gradually emerged in the field of computer vision and has become an important research field. Specifically, semantic segmentation achieves image classification through pixel-level segmentation to identify the category to which any pixel in the image belongs, to achieve deep feature extraction of the image semantic level, and ultimately obtain semantic information in the image.
[0004] Drone aerial images have greatly enriched the data source due to their high resolution and flexible acquisition methods. Through semantic segmentation, ground targets such as buildings, roads, vegetation, and water bodies can be accurately identified and classified. This not only helps to improve the accuracy and efficiency of urban planning, but also plays an important role in post-disaster rescue, environmental monitoring, and agricultural management. For example, in urban planning, semantic segmentation can help planners better understand land use and optimize urban layout; in the agricultural field, it can be used to monitor the growth of crops and improve the level of intelligence in agricultural production.
[0005] In addition, semantic segmentation of drone aerial images also plays a key role in natural resource management. By accurately marking natural elements such as forests, grasslands, and water resources, it can help relevant departments conduct resource assessment and environmental protection. In the construction of smart cities, real-time updates and analysis of urban dynamic information can assist in decision-making in traffic management, pollution monitoring, and public safety. For example, in the field of public safety, when natural disasters such as earthquakes, floods, and fires occur, drones can quickly take off to take aerial photos of the affected areas. Semantic segmentation technology can help quickly identify and distinguish different types of land cover, damaged buildings, trapped people, etc., provide emergency management departments with instant disaster assessment reports, and guide the effective allocation of rescue resources.
[0006] Semantic segmentation technology can be divided into traditional semantic segmentation methods and semantic segmentation methods based on deep learning. Traditional semantic segmentation methods are different from most of the deep convolutional network algorithms currently used. They mainly segment and identify different objects in the image based on artificial features designed such as texture, grayscale, and geometric features, so as to separate the objects from the image background. This type of method usually requires manual adjustment of algorithm parameters, which is time-consuming and labor-intensive, and the model effect and accuracy are inferior to those based on deep learning methods. Traditional methods include threshold segmentation, edge segmentation, and region construction. With the widespread application of deep learning technology, semantic segmentation methods based on deep learning have made breakthrough progress. Compared with traditional methods, their segmentation effects are better and more robust, such as DeepLab, PSPNet, SegFormer and other deep network models for semantic segmentation.
[0007] In the past decade, semantic segmentation technology based on RGB visible light images has made great progress, with significant improvements in speed and accuracy. However, due to the characteristics of RGB visible light image sensors, it is difficult to obtain clear images in dim light and low visibility environments, such as at night or on rainy days, which greatly limits the performance of related methods. In recent years, infrared thermal imaging cameras have attracted widespread attention because they are not affected by lighting and weather conditions and can provide additional and effective supplementary information in visible light-restricted scenes. Based on this, the development of semantic segmentation network models for RGB-T images (i.e., visible light and thermal infrared images) has important practical significance.
[0008] The current methods involving RGB-T image semantic segmentation include: Invention patent (application number: 202110187778.8, name: A RGB-T image semantic segmentation method based on modal difference reduction) utilizes the modal difference between visible light features and thermal infrared features to reduce and fuse them in the network coding stage, and at the same time mines the multi-scale contextual information of the fused features and their long-range dependencies. However, since this method uses a modal difference reduction method, complex multimodal fusion methods may make the model difficult to interpret. It becomes difficult to understand how the model exchanges information and makes decisions between different modalities, which is a problem for certain application scenarios that require high interpretability; the invention patent (application number: 202310606344.6, name: A semantic segmentation method based on single-stream cross-modal fusion of RGB-T images) uses a single-stream cross-modal fusion network to process two modal images, visible light and thermal infrared images, combined with a feature classification processing module and a multi-head parallel decoder supervision network to solve the problems of large number of parameters inherent in the RGB-T dual-stream network. However, since the system deploys a single-stream framework, the single-stream framework will cause relevant information loss compared to the dual-stream framework. This information loss will cause poor feature expression capabilities. The invention patent (application number: 202311049781.9, name: Image semantic segmentation algorithm and application based on multimodal feature fusion) uses the collaborative method of the federated learning concept to deliver the extraction task of a single modal feature to different extraction modules for independent completion, and uses the fusion module to check the completion of the single modal feature extraction task. After determining that all the extraction tasks are completed, the extracted single modal features are fused to obtain multimodal fusion features. Finally, this feature is used as input and the ASPP module is used to realize image semantic segmentation prediction. Due to the high computational consumption of federated learning, this is a heavy burden for edge devices.
[0009] In view of this, due to the complexity of real-world applications, how to effectively fuse the information of visible light images and thermal infrared images to achieve a semantic segmentation method that takes into account both accuracy and efficiency is an urgent problem that needs to be solved. Summary of the invention
[0010] In order to overcome the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method, system, device and medium for semantic segmentation of aerial images, which mainly solves the problem that the prior art does not take into account the modal differences between visible light images and thermal infrared images, which leads to low semantic segmentation accuracy. A convolutional network model based on an attention mechanism is designed for the semantic segmentation of RGB-T images. The model uses a two-stream convolutional neural network encoder to perform feature extraction and complementary perception fusion on the RGB-T images obtained by drone aerial photography, and captures the weights of different modal data in different channels through a channel attention module and a spatial attention module, and improves the spatial resolution and positioning accuracy, so that the RGB-T image semantic segmentation model has the characteristics of high image data feature utilization and good semantic segmentation accuracy under different lighting conditions.
[0011] A method for semantic segmentation of aerial images, characterized by the following steps:
[0012] Step 1: Obtain visible light RGB images and thermal infrared images, perform preprocessing, construct the RGB-T semantic segmentation original dataset, and divide it into training set and test set in proportion; then perform random mask processing to generate random masks for each modality. The generated dataset is called RGB-T semantic segmentation compensation dataset;
[0013] For each random mask M of each modality, define its complementary mask If a pixel in the original mask is selected, its pixel is set to 0, and the corresponding pixel in the complementary mask is not selected, and its pixel is set to 1; the image block is processed using the above random mask and complementary mask, and for the visible light modality, formula (1) is applied for processing, and for the thermal infrared modality, formula (2) is applied for processing;
[0014]
[0015] in is the RGB result image obtained after mask processing, is the thermal infrared result image obtained after mask processing. and is the original RGB image and the image after background filling, and It is the original thermal infrared image and the image after background filling;
[0016] Step 2: For the images in the training set of the RGB-T semantic segmentation original dataset and the RGB-T semantic segmentation compensation dataset in step 1, use the RGB encoder and thermal infrared encoder to extract the feature map F respectively. rgb and feature map F thr , through global average pooling fGAP from F rgband F thr The global context information is extracted and further converted into channel attention weights R and T through the multi-layer perceptron fMLP. R and T correspond to the channel attention weights of visible light and thermal infrared features respectively, and the visible light modal features and thermal infrared modal features are obtained. The features of the two modalities are combined, and the information from different modalities is combined by feature splicing to obtain the fused feature map.
[0017] Step 3: Input the fused feature map into the feature extraction module. The channel attention weights of visible light and thermal infrared are directly applied to the feature maps of the corresponding modalities to enhance the features of each modality. On the basis of feature enhancement, the weighted features of the two modalities are fused to generate a global feature representation map that combines the advantages of the two modalities, and extract high-level semantic features: for RGB features, apply formula (3) for processing, and for thermal infrared features, apply formula (4) for processing;
[0018]
[0019] where F r ' gb represents the visible light modal feature map enhanced by the feature extraction module, F t ' hr represents the thermal infrared modal feature map enhanced by the feature extraction module, σ is the sigmoid function, softmax is applied to the combination of R and T, [·;·] represents the column stacking operation, R and T represent the channel attention weights of visible light and thermal infrared, respectively, and F rgb and F thr Represents the original visible light modal feature map and the original thermal infrared modal feature map;
[0020] Step 4: Input the high-level semantic features and other high-level features of the encoding stream into the feature fusion module, use element-by-element multiplication to achieve feature fusion, integrate the features adjusted by the attention mechanism, and use the convolution / non-linear processing layer to refine and encode the fused features, and finally output the enhanced RGB features and thermal infrared features, which are passed to the decoder: For RGB features, apply formula (5) for processing, and for thermal infrared features, apply formula (6) for processing;
[0021] F r ' gb =σ(Fusion rgb (R⊙F rgb ,T⊙F rgb )) (5)
[0022] F t ' hr =σ(Fusionthr (R⊙F thr ,T⊙F thr )) (6)
[0023] Among them, F r ' gb represents the visible light modality feature map enhanced by the feature fusion module, F t ' hr represents the thermal infrared modal feature map enhanced by the feature fusion module, σ is the sigmoid function, and Fusion rgb Represents the fusion function for visible light image features, Fusion thr represents the fusion function for thermal infrared image features, ⊙ represents element-by-element multiplication, R and T represent the channel attention weights of visible light and thermal infrared light, and F rgb represents the original visible light modal feature map, F thr Represents the original thermal infrared modal characteristic diagram;
[0024] Step 5: The decoder uses the RGB features and thermal infrared features passed down from the feature fusion module to gradually upsample and restore the image resolution, and finally generates a semantic segmentation prediction map;
[0025] Step 6: Loss function calculation: define the set of semantic segmentation prediction maps generated by the RGB-T semantic segmentation original dataset as the first set, and define the set of semantic segmentation prediction maps generated by the RGB-T semantic segmentation compensated dataset as the second set; calculate the loss function between the first set and the second set to measure the difference between the prediction result and the true label;
[0026] Step 7: Update model parameters: Update the weight vector and bias term of the convolutional neural network through back propagation and optimizer, and iteratively optimize the model parameters;
[0027] Step 8: Input the image to be segmented, and input the image to be semantically segmented into the trained convolutional neural network classification training model; obtain the semantic segmentation image predicted by the model output.
[0028] The aerial image semantic segmentation system comprises the following modules:
[0029] Image acquisition module: used to acquire visible light images and thermal infrared images of drone aerial photography scenes;
[0030] Image processing module: used for preprocessing and random mask method to process the acquired images;
[0031] A feature extraction module is used to extract features from the visible light image and the thermal infrared image through an encoder to obtain visible light features and thermal infrared modal features; wherein the visible light features and the thermal infrared modal features both include multi-level modal features;
[0032] Feature fusion module: used for inputting the visible light modal feature and the thermal infrared modal feature into the feature fusion module for fusion feature calculation to generate a fused feature map;
[0033] Comprehensive attention extraction module: used to obtain high-level semantic features and high-level information features through the channel attention mechanism and spatial attention mechanism of the feature map;
[0034] Encoding module: used to input the fusion features of each level into the encoder for encoding processing;
[0035] A random mask compensation module: used for selecting one of the first semantic segmentation image and the second semantic segmentation image as the semantic segmentation image of the target RGB-T image pair according to the texture information missing of the target RGB-T image pair;
[0036] Result generation module: used to generate the final segmentation results.
[0037] A computer device comprises: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, a RGB-T image semantic segmentation method for drone aerial photography applications is performed.
[0038] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a RGB-T image semantic segmentation method for unmanned aerial photography applications is executed.
[0039] By adopting the above technology, compared with the prior art, the beneficial effects of the present invention are as follows:
[0040] 1) The network of the present invention uses dual-modal feature fusion, and extracts feature information from dual streams of visible light and thermal infrared, making the feature information richer and better coping with the loss of texture signals of single-modal data;
[0041] 2) The multi-level modal feature extraction of the network of the present invention fully extracts and utilizes the information of each modality. By extracting and enhancing high-level feature information, the performance of the model in the RGB-T semantic segmentation task is improved, enabling the model to more accurately understand and segment different objects in complex scenes;
[0042] 3) The present invention uses a random masking method to force the model to handle more image variants during training, including cases where some information is lost. This not only improves the generalization ability of the model, but also enhances the performance of the model in the face of various interferences and occlusions in the real world. Through this data augmentation technology, the model can better adapt to diverse inputs and improve the accuracy and stability of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required to be used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative labor.
[0044] Figure 1 It is a schematic diagram of the dual-branch RGB-T semantic segmentation network structure provided by the present invention;
[0045] Figure 2 It is a structural schematic diagram of the mask operation provided by the present invention;
[0046] Figure 3 It is a structural schematic diagram of the semantic segmentation system provided by the present invention. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0048] Implementation example 1:
[0049] This embodiment discloses a method for semantic segmentation of aerial images, including:
[0050] Step 1: Construct the original RGB-T semantic segmentation dataset and divide it into a training set and a test set: Obtain visible light (RGB) images and thermal infrared (Thermal) images from the visible light camera and infrared thermal imager carried by the drone, preprocess these RGB-T images, and perform pixel-level semantic segmentation annotation on these RGB-T images. The preprocessing operations are as follows:
[0051] Step 1-1: Histogram equalization: Improve the contrast of the image and make the brightness distribution of the image more uniform; this is especially important for thermal infrared images because the brightness distribution of thermal infrared images will be affected by the ambient temperature;
[0052] Step 1-2: Noise removal: remove noise from the image, such as Gaussian noise and salt and pepper noise; noise will affect the accuracy of feature extraction, so it needs to be removed;
[0053] Step 1-3: Scaling: Scale both images to the same resolution; since visible light images and thermal infrared images are taken by different devices at different distances, their resolutions will be different in this case;
[0054] Step 1-4: Rotation correction: If there is rotation between the two images, they can be adjusted to the same orientation through rotation correction;
[0055] Optionally, since the pixel-level semantic segmentation annotation of RGB-T image pairs is very expensive, data augmentation can be performed;
[0056] Optionally, you can use open source RGB-T semantic segmentation datasets, such as the MFNet dataset, PST900 dataset, and KP dataset;
[0057] Step 2: Generate a random mask for each modality. The dataset generated by the mask is called the RGB-T semantic segmentation compensation dataset. This mask is a two-dimensional matrix with the same size as the image, in which the element value indicates whether the corresponding pixel is masked, such as Figure 2 As shown in the figure, a mask is formed in some areas of the figure; 1 in the mask means retaining the original information, and 0 indicates that the position needs to be masked; multiple masking strategies, including: random square mask: at a random position, a square area is randomly generated with half the height and width of the image as the side length for masking; random block mask: half of the image is randomly selected (with a ratio of 0.5) and masked with blocks of different sizes (for example, 8x8, 16x16, 32x32, 64x64 pixels); for each pair of RGB-T images, the generated masks are complementary; this means that if a part of the RGB image is covered by the mask, the corresponding thermal infrared part remains clear, and vice versa, which ensures that the network obtains as much information as possible from each modality while handling partial missing situations; for the random mask M of each modality, its complementary mask is defined This means that if a pixel in the original mask is selected (set to 0), the corresponding pixel in the complementary mask is not selected (set to 1), and vice versa. The image block is processed using the above mask and the complementary mask. For the visible light modality, the formula is applied:
[0058]
[0059] in is the RGB result image obtained after mask processing, and is the original RGB image and the image after background filling; similarly, for the thermal infrared modality, the formula applies:
[0060]
[0061] in is the thermal infrared result image obtained after mask processing. and It is the original thermal infrared image and the image after background filling;
[0062] Step 3: For the images in the training set of the RGB-T semantic segmentation original dataset and the RGB-T semantic segmentation compensation dataset, the RGB encoder and the thermal infrared encoder independently extract the feature map F rgb and F thr , these feature maps retain the spatial structure and high-level semantic information of the original image; then, global average pooling (fGAP) is used to obtain the rgb and F thr The global context information is extracted and further converted into channel attention weights R and T through a multi-layer perceptron (fMLP), which correspond to the weights of RGB and thermal infrared features respectively;
[0063] Step 4: For the feature extraction module, by using the weighted sum of the attention mechanism To generate complementary attention weights, a nonlinear activation function (σ) is then used to enhance the model’s expressiveness; here σ is the sigmoid function, [·;·] represents a column-by-column stacking operation, and softmax is applied to the combination of R and T to ensure that they form a probability distribution before fusion, thereby reflecting complementarity; for RGB features, the enhancement formula is:
[0064]
[0065] For thermal infrared features, the enhancement formula is:
[0066]
[0067] Step 5: For the feature fusion module, the product of R (channel attention for visible light) and F (feature map) and the product of T (channel attention for thermal infrared light) and F (feature map) are calculated by element-wise multiplication (⊙), and then the scaled attention weights are processed using the sigmoid nonlinear activation function (σ) and applied to the original RGB features and thermal infrared features; this forms the enhanced feature F r ' gb and F t ' hr, respectively represent the attention-guided RGB features and thermal infrared features; for RGB features, the enhancement formula is:
[0068] F r ' gb =σ(Fusion rgb (R⊙F rgb ,T⊙F rgb )) (5)
[0069] For thermal infrared features, the enhancement formula is:
[0070] F t ' hr =σ(Fusion thr (R⊙F thr ,T⊙F thr )) (6)
[0071] Among them, Fusion rgb and Fusion thr They represent the fusion functions for RGB and thermal infrared image features, respectively. They can be the same or different according to the data characteristics, aiming to integrate the attention-weighted features provided by R and T;
[0072] Step 6: The decoder usually starts with multiple feature maps of different scales, which come from different levels of the encoder, such as Figure 1 As shown in the figure, each layer operates on the generated feature map; in order to restore the spatial resolution of the feature map and maintain the details, the decoder uses upsampling operations (such as bilinear interpolation and transposed convolution); in addition, a skip connection is introduced, that is, the high-resolution features of the corresponding encoder layer are added to the upsampled features to retain low-level local details;
[0073] Step 7: Use the masked image blocks to train the network and adjust the network weights through back propagation to minimize the total loss and improve network performance;
[0074] Step 8: Input the image to be segmented, and input the image to be semantically segmented into the trained convolutional neural network classification training model; obtain the semantic segmentation image predicted by the model output.
[0075] Implementation Example 2:
[0076] The purpose of this embodiment is to provide an aerial image semantic segmentation system. Figure 3 As shown, including:
[0077] Image acquisition module: used to acquire visible light images and thermal infrared images of drone aerial photography scenes;
[0078] Image processing module: used for preprocessing and random mask method to process the acquired images;
[0079] A feature extraction module is used to extract features from the visible light image and the thermal infrared image through an encoder to obtain visible light features and thermal infrared modal features; wherein the visible light features and the thermal infrared modal features both include multi-level modal features;
[0080] Feature fusion module: used for inputting the visible light modal feature and the thermal infrared modal feature into the feature fusion module for fusion feature calculation to generate a fused feature map;
[0081] Comprehensive attention extraction module: used to obtain high-level semantic features and high-level information features through the channel attention mechanism and spatial attention mechanism of the feature map;
[0082] Encoding module: used to input the fusion features of each level into the encoder for encoding processing and return them to the decoder;
[0083] A random mask compensation module: used for selecting one of the first semantic segmentation image and the second semantic segmentation image as the semantic segmentation image of the target RGB-T image pair according to the texture information missing of the target RGB-T image pair;
[0084] Result generation module: used to generate the final segmentation results.
[0085] The steps involved in the apparatus of the above implementation example 2 correspond to the method implementation example 1. For the specific implementation method, please refer to the relevant description part of the implementation example 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method of the present invention.
[0086] Implementation example three:
[0087] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method of implementation example 1;
[0088] When it comes to actual deployment, the drone's processor core uses highly specialized hardware to meet the stringent requirements of the mission. Application-specific integrated circuits (ASICs) are used to achieve highly customized data processing capabilities with low power consumption; digital signal processors (DSPs) are selected to accelerate image processing and signal analysis tasks; digital signal processing devices (DSPDs), programmable logic devices (PLDs) and field programmable gate arrays (FPGAs) are also suitable for processing complex drone control logic and image processing algorithms due to their flexibility and high performance.
[0089] In addition, using a microcontroller unit (MCU) or microprocessor as the core processor can ensure the stable operation of the system and basic task management, while high-end microprocessors can support complex artificial intelligence algorithms, enabling drones to have the ability to autonomously learn and adapt to environmental changes.
[0090] Implementation example 4:
[0091] This embodiment further provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the method of implementation example 1 is implemented;
[0092] The system needs to use a combination of the following storage technologies to achieve ideal performance:
[0093] Cache storage: Static random access memory (SRAM) is used as a temporary cache to quickly process and temporarily store the high-resolution video stream being recorded. SRAM has a fast access speed and can effectively reduce the delay of data processing, which is particularly important for scenarios that require instant response.
[0094] Non-volatile solid-state storage: Flash memory (such as solid-state drives SSD) is used as the main data storage medium to ensure that the collected data will not be lost even if the drone suddenly loses power. SSD has the characteristics of fast read and write speed and good shock resistance, which is suitable for drones to perform long-term, high-intensity aerial photography tasks in complex and changing environments.
[0095] Remote cloud storage and real-time transmission: By integrating 4G / 5G communication modules, drones can transmit key data to cloud servers in real time, enabling remote monitoring and instant analysis. This strategy not only reduces the pressure on onboard storage, but also enables the ground control center to obtain inspection results in the first place and make timely decision responses.
[0096] Backup and fault-tolerance mechanism: Programmable read-only memory (PROM), EEPROM or EPROM technology is used to provide permanent storage for important system parameters and flight logs. These memories have certain data modification capabilities (such as EEPROM and EPROM can be erased and rewritten in a specific way) to ensure the long-term preservation of key information and system reliability.
Claims
1. A method for semantic segmentation of aerial images, characterized in that: The steps include: Step 1: Obtain visible light RGB images and thermal infrared images, perform preprocessing, construct the RGB-T semantic segmentation original dataset, and divide it into training set and test set in proportion; then perform random mask processing to generate random masks for each modality. The generated dataset is called RGB-T semantic segmentation compensation dataset; For each random mask M of each modality, define its complementary mask If a pixel in the original mask is selected, its pixel is set to 0, and the corresponding pixel in the complementary mask is not selected, and its pixel is set to 1; the image block is processed using the above random mask and complementary mask, and for the visible light modality, formula (1) is applied for processing, and for the thermal infrared modality, formula (2) is applied for processing; in is the RGB result image obtained after mask processing, is the thermal infrared result image obtained after mask processing. and is the original RGB image and the image after background filling, and It is the original thermal infrared image and the image after background filling; Step 2: For the images in the training set of the RGB-T semantic segmentation original dataset and the RGB-T semantic segmentation compensation dataset in step 1, use the RGB encoder and thermal infrared encoder to extract the feature map F respectively. rgb and feature map F thr , through global average pooling fGAP from F rgb and F thr The global context information is extracted and further converted into channel attention weights R and T through the multi-layer perceptron fMLP. R and T correspond to the channel attention weights of visible light and thermal infrared features respectively, and the visible light modal features and thermal infrared modal features are obtained. The features of the two modalities are combined, and the information from different modalities is combined by feature splicing to obtain the fused feature map. Step 3: Input the fused feature map into the feature extraction module. The channel attention weights of visible light and thermal infrared are directly applied to the feature maps of the corresponding modalities to enhance the features of each modality. On the basis of feature enhancement, the weighted features of the two modalities are fused to generate a global feature representation map that combines the advantages of the two modalities, and extract high-level semantic features: for RGB features, apply formula (3) for processing, and for thermal infrared features, apply formula (4) for processing; where F r ' gb represents the visible light modal feature map enhanced by the feature extraction module, F t ' hr represents the thermal infrared modal feature map enhanced by the feature extraction module, σ is the sigmoid function, softmax is applied to the combination of R and T, [·;·] represents the column stacking operation, R and T represent the channel attention weights of visible light and thermal infrared, respectively, and F rgb and F thr Represents the original visible light modal feature map and the original thermal infrared modal feature map; Step 4: Input the high-level semantic features and other high-level features of the encoding stream into the feature fusion module, use element-by-element multiplication to achieve feature fusion, integrate the features adjusted by the attention mechanism, and use the convolution / non-linear processing layer to refine and encode the fused features, and finally output the enhanced RGB features and thermal infrared features, which are passed to the decoder: For RGB features, apply formula (5) for processing, and for thermal infrared features, apply formula (6) for processing; F r ′ gb =σ(Fusion rgb (R⊙F rgb ,T⊙F rgb )) (5) F t ′ hr =σ(Fusion thr (R⊙F thr ,T⊙F thr )) (6) Among them, F r ' gb represents the visible light modality feature map enhanced by the feature fusion module, F t ' hr represents the thermal infrared modal feature map enhanced by the feature fusion module, σ is the sigmoid function, and Fusion rgb Represents the fusion function for visible light image features, Fusion thr represents the fusion function for thermal infrared image features, ⊙ represents element-by-element multiplication, R and T represent the channel attention weights of visible light and thermal infrared light, and F rgb represents the original visible light modal feature map, F thr Represents the original thermal infrared modal characteristic diagram; Step 5: The decoder uses the RGB features and thermal infrared features passed down from the feature fusion module to gradually upsample and restore the image resolution, and finally generates a semantic segmentation prediction map; Step 6: Loss function calculation: define the set of semantic segmentation prediction maps generated by the RGB-T semantic segmentation original dataset as the first set, and define the set of semantic segmentation prediction maps generated by the RGB-T semantic segmentation compensated dataset as the second set; calculate the loss function between the first set and the second set to measure the difference between the prediction result and the true label; Step 7: Update model parameters: Update the weight vector and bias term of the convolutional neural network through back propagation and optimizer, and iteratively optimize the model parameters; Step 8: Input the image to be segmented, and input the image to be semantically segmented into the trained convolutional neural network classification training model; obtain the semantic segmentation image predicted by the model output.
2. A system for semantic segmentation of aerial images, characterized in that: Contains the following modules: Image acquisition module: used to acquire visible light images and thermal infrared images of drone aerial photography scenes; Image processing module: used for preprocessing and random mask method to process the acquired images; A feature extraction module is used to extract features from the visible light image and the thermal infrared image through an encoder to obtain visible light features and thermal infrared modal features; wherein the visible light features and the thermal infrared modal features both include multi-level modal features; Feature fusion module: used for inputting the visible light modal feature and the thermal infrared modal feature into the feature fusion module for fusion feature calculation to generate a fused feature map; Comprehensive attention extraction module: used to obtain high-level semantic features and high-level information features through the channel attention mechanism and spatial attention mechanism of the feature map; Encoding module: used to input the fusion features of each level into the encoder for encoding processing; A random mask compensation module: used for selecting one of the first semantic segmentation image and the second semantic segmentation image as the semantic segmentation image of the target RGB-T image pair according to the texture information missing of the target RGB-T image pair; Result generation module: used to generate the final segmentation results.
3. A computer device comprising: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, a RGB-T image semantic segmentation method for drone aerial photography applications is performed.
4. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, executes a RGB-T image semantic segmentation method for drone aerial photography applications.
Citation Information
Patent Citations
RGB-T image semantic segmentation method based on modal difference reduction
CN112991350A
Semantic segmentation method based on RGB-T image uniflow cross-modal fusion
CN116778155A
Image semantic segmentation algorithm based on multi-modal feature fusion and application
CN117152431A
RGB-T image semantic segmentation method and device
CN116091765A
Identification and segmentation method for high-brightness information of night vision halation image
CN117078723A
Cited By
Multi-modal target detection method based on dynamic illumination perception random mask
CN120259294A
A Multimodal Object Detection Method Based on Dynamically Light-Aware Random Masking
CN120259294B
Agricultural meteorological disaster time sequence prediction system and method based on multi-modal data fusion
CN120559760A
Agrometeorological disaster time series prediction system and method based on multimodal data fusion
CN120559760B
Low-altitude multi-modal image fusion method based on semantic prompt guidance
CN121074567A
