A semantic segmentation method, device and storage medium based on multispectral image
Through the category-spectral correlation module and the spectral channel enhancement module, supervised learning and channel attention mechanisms are used to solve the problems of large intra-class differences and small inter-class differences in multi-spectral images, and more accurate semantic segmentation results are achieved.
Patent Information
- Application Number
- CN202210533579.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-17
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2042-05-17
AI Technical Summary
The existing multispectral image semantic segmentation method has failed to effectively solve the problems of large in-class differences and small inter-class differences, and has failed to make full use of multispectral features, resulting in blurred segmentation results and noise interference.
Using the category-spectral correlation module and the spectral channel enhancement module, through supervised learning and channel attention mechanism, improve the similarity between pixels in the same category, reduce the differences between classes, reduce the redundant information in multi-spectral features, and increase the class. difference between.
It effectively improves the accuracy of semantic segmentation results, reduces intra-class differences and noise interference, enhances inter-class differences, and improves the segmentation performance of multi-spectral images.
Smart Images
Figure CN115082492B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a semantic segmentation method, device and storage medium based on multispectral images. Background Art
[0002] In recent years, with the rapid increase of surface observation data of high-altitude scenes, semantic segmentation has been widely used in high-resolution remote sensing images. This task usually only requires the input of RGB images, and learns semantic expressions from rich texture information to obtain good results. However, these methods only analyze objects from a single image perspective, that is, they only distinguish different categories of objects through visible light texture information, resulting in certain limitations of the model.
[0003] In order to solve the above problems, existing methods introduce multispectral images, which integrate visual information of light at different wavelengths to compensate for the defects of RGB images affected by factors such as lighting, thereby improving model performance. However, most existing methods simply fuse the information of visible light and non-visible light, without considering that pixels of the same category have similar multispectral features, so it is difficult to solve the problem of large intra-class differences. In addition, some methods only introduce category context information in the spatial dimension, without considering the different degrees of redundancy between multispectral features in different categories, making it difficult to solve the interference and noise caused by redundant information, and there is a problem of small inter-class differences. Summary of the invention
[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the object of the present invention is to provide a semantic segmentation method, device and storage medium based on multispectral images.
[0005] The technical solution adopted by the present invention is:
[0006] A semantic segmentation method based on multispectral images comprises the following steps:
[0007] Collect and annotate multispectral datasets for semantic segmentation;
[0008] Build a semantic segmentation model;
[0009] Using the multispectral dataset to train the semantic segmentation model;
[0010] Acquire an image to be processed, input the image to be processed into the trained semantic segmentation model, and output a semantic segmentation result;
[0011] The semantic segmentation model includes a category-spectrum association module, which is used to obtain continuous and accurate segmentation results by increasing the similarity between pixels of the same category and reducing the difference between categories.
[0012] Furthermore, the semantic segmentation model also includes a spectral channel enhancement module;
[0013] The spectral channel enhancement module is used to utilize the channel attention mechanism to focus on important information in the multi-spectral features, so as to reduce redundant information of different categories in the multi-spectral features.
[0014] Furthermore, the collecting and labeling of multispectral datasets for semantic segmentation includes:
[0015] Collecting multispectral images from high-altitude scenes and annotating the multispectral images to construct a multispectral dataset;
[0016] The annotated multispectral dataset is divided into three parts: training set, validation set, and test set;
[0017] For the high-resolution images in the training set, fixed-size images are randomly captured, and for the high-resolution images in the validation set and the test set, fixed-size images are captured using a sliding window.
[0018] Furthermore, the semantic segmentation model also includes an encoder, a spectral channel enhancement module and a decoder;
[0019] The encoder is used to extract features from the multispectral image;
[0020] The category-spectrum association module is used to obtain a preliminary segmentation result in a supervised manner, and to perform soft category mean pooling on multi-spectral features to obtain a category-spectrum relationship matrix, thereby reducing intra-category differences;
[0021] The spectral channel enhancement module is used to calculate the channel attention score of each category, assign weights to the feature channels of different category areas, reduce the redundant information of different categories in multi-spectral features, and thus increase the difference between categories;
[0022] The decoder is used to decode the multispectral features and output semantic segmentation results.
[0023] Furthermore, define X h is the high-level feature after the first upsampling in the decoder; define X l The low-level features output by the first stage of the backbone network in the encoder;
[0024] The workflow of the category-spectrum association module is as follows:
[0025] The high-level feature X h The number of channels is reduced to the number of categories N, and a softmax operation is performed on the category dimension to obtain N attention maps as the category attention map X p ;
[0026] Reduce the low-level features X l The number of channels is , and feature X is obtained;
[0027] Attention map X for the category p The pixel features belonging to different category areas in are respectively average pooled to obtain N multispectral features, and the category-spectrum relationship matrix M is formed by combining the features X.
[0028] Furthermore, the workflow of the spectral channel enhancement module is as follows:
[0029] The category-spectrum relationship matrix M is regarded as N independent spectral features, and based on the channel attention mechanism, N channel attention scores are calculated respectively to form a spectral channel attention score A;
[0030] For the low-level feature X l Perform feature extraction and obtain feature X v ;
[0031] According to the spectral channel attention score A and the feature X v , for the category attention map X p The pixel features in the corresponding area of category k are redistributed with channel weights and combined with the low-level features X l Spliced together, we get the output feature Y.
[0032] Further, the encoder includes a backbone network and an ASPP module;
[0033] The backbone network adopts dilated convolution instead of downsampling operation to increase the receptive field while preventing the resolution from decreasing.
[0034] The ASPP module is used to fuse features extracted by multiple convolutional layers with different expansion rates to extract multi-scale context features.
[0035] Furthermore, the adopting the multispectral dataset to train the semantic segmentation model includes:
[0036] The cross entropy function is used to calculate the loss, and the stochastic gradient descent algorithm is used to update the parameters of the network until convergence.
[0037] Another technical solution adopted by the present invention is:
[0038] A semantic segmentation device based on multispectral images, comprising:
[0039] at least one processor;
[0040] at least one memory for storing at least one program;
[0041] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.
[0042] Another technical solution adopted by the present invention is:
[0043] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to execute the method described above when executed by the processor.
[0044] The beneficial effect of the present invention is that the present invention can better extract complementary information from multispectral images by improving the similarity between pixels of the same category and reducing the difference between categories, thereby solving the problem of large intra-category differences. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0046] Figure 1 is a flowchart of a semantic segmentation method based on multispectral images in an embodiment of the present invention;
[0047] Figure 2 is a structural schematic diagram of a semantic segmentation model based on multispectral images in a high-altitude scene in an embodiment of the present invention;
[0048] Figure 3 is a schematic diagram of a category-spectrum association module in an embodiment of the present invention;
[0049] Figure 4 is a schematic diagram of a spectral channel enhancement module in an embodiment of the present invention;
[0050] Figure 5 This is an example diagram of incorrect predictions in which the intra-class difference is large and the inter-class difference is small in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0052] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0053] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0054] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0055] The existing semantic segmentation methods based on multispectral images have the following problems: (1) They do not take into account that pixels of the same category have similar multispectral features, which leads to ambiguous semantic expression and large intra-class differences, such as Figure 5 (a) As shown in Figure 2. (2) The redundant information between visible light images and non-visible light images is not taken into account. Only the category context information is introduced in the spatial dimension, which makes it difficult to solve the interference and noise caused by redundancy. There is a problem of small inter-class differences, such as Figure 5(b) As shown. In response to problem (1), the present invention spatially divides pixels into different category areas, and extracts common multispectral features for pixels of the same category. On the one hand, the present invention uses a supervised approach to obtain a rough segmentation result, prompting the model to correctly divide pixel features; on the other hand, it considers similar pixels at long distances or with large appearance differences, enhances the semantic expression of pixel features within the category and reduces intra-category differences. Finally, the present invention extracts multispectral features for each category, that is, constructs the relationship between category and spectrum. In response to problem (2), the present invention uses a channel attention mechanism to reduce the redundant information of multispectral features in different categories. Since different categories pay attention to different spectral feature channels, the present invention calculates channel attention scores for different categories, uses different channel attention scores for pixels in different category areas, reduces the impact of noise in multispectral features according to different categories, and thereby increases inter-class differences.
[0056] like Figure 1 As shown, this embodiment provides a semantic segmentation method based on multispectral images, which specifically includes the following steps:
[0057] S1. Collect and annotate multispectral datasets for semantic segmentation.
[0058] As an optional implementation, step S1 includes the following steps S11-S13:
[0059] S11. Collect multispectral images from high-altitude scenes and annotate the multispectral images to construct a multispectral dataset, wherein the images include six categories: "road", "building", "tree", "car", "low plant", and "debris".
[0060] S12. Divide the annotated multispectral dataset into three parts: training set, validation set, and test set.
[0061] S13. For the high-resolution images in the training set, images of a fixed size (512*512 pixels) are randomly captured. For the high-resolution images in the validation set and the test set, images of a fixed size (512*512 pixels) are captured using a sliding window.
[0062] S2. Build a semantic segmentation model. For the problem of large intra-class differences, use a supervised approach to divide spatial pixels into categories and extract common features from pixels of the same category, thereby explicitly establishing the connection between categories and spectra; for the problem of small inter-class differences, use the channel attention mechanism to assign weights to feature channels of different category regions, allowing the network to let the model autonomously focus on useful information and reduce the impact of noise.
[0063] In high-altitude scenes, the semantic segmentation model extracts features from the input multispectral image, assigns a category label to each pixel, and finally obtains the semantic segmentation result. Figure 2 As shown in the figure, the semantic segmentation model mainly consists of four parts: (1) Encoder: It aims to extract features from multispectral images; (2) Category-spectral association module: It obtains preliminary segmentation results in a supervised manner, and performs soft category mean pooling on the multispectral features to obtain the category-spectral relationship matrix, thereby reducing the intra-class differences; (3) Spectral channel enhancement module: It calculates the channel attention score of each category, assigns weights to the feature channels of different category areas, reduces the redundant information of different categories in the multispectral features, and thereby increases the inter-class differences; (4) Decoder: It decodes the multispectral features and outputs the semantic segmentation results.
[0064] As an optional implementation, the encoder works as follows: the multi-channel multispectral image I spec Input to the encoder to extract features. The encoder consists of two parts: a backbone network and an ASPP module. The backbone network is a ResNet101 model pre-trained on the ImageNet dataset. Since the ResNet model uses 5 downsamplings, the resolution of its output features is 1 / 32 of the input image, that is, the output span is 32, which loses a lot of detail information and causes the model performance to degrade. Therefore, the backbone network uses dilated convolution instead of downsampling operations to prevent resolution degradation while increasing the receptive field. Specifically, the downsampling operations in the last two residual blocks are replaced by dilated convolutions with expansion rates of 2 and 4, respectively, so that the output span of the network is 8. In addition, the present invention uses the ASPP module to fuse the features extracted by multiple convolutional layers with different expansion rates to extract multi-scale context features.
[0065] As an optional implementation, the category-spectrum association module works as follows: First, define X h It is the high-level feature after the first upsampling in the decoder, which has rich category information. Define X l It is the low-level feature output by the first stage of the backbone network in the encoder, which has rich detailed information. This module is divided into two steps: (1) Generate the category attention map X p Through supervision, the network learns the attention map of each category, that is, the probability that each pixel belongs to that category. (2) Calculate the category-spectrum relationship matrix M. Based on the self-attention mechanism, the relationship between the category and the spectrum is obtained through matrix operation.
[0066] See also Figure 3 , generate category attention map X p The specific steps are as follows: In this embodiment, the high-level semantic feature X is transformed into hThe number of channels is reduced to the number of categories N, and a softmax operation is performed on the category dimension to obtain N attention maps, namely the category attention map X p In order to better learn the relationship between categories and spectra, the present invention adopts supervised learning in the training stage and introduces a loss function to make X p Close to semantic segmentation labels.
[0067] See also Figure 3 , the steps for calculating the category-spectrum relationship matrix M are as follows: reduce the multi-spectral feature X through a convolution layer with a convolution kernel size of 1x1 l The number of channels is calculated to obtain feature X. In order to extract the semantic expression obtained, the present invention extracts the feature commonality of pixels of the same category and integrates the category information into the multispectral feature, thereby reducing the intra-category difference. Specifically, soft category mean pooling is adopted, that is, X is p The pixel features belonging to different category areas in are mean pooled respectively to obtain N multi-spectral features and form a category-spectrum relationship matrix M.
[0068] As an optional implementation, the spectral channel enhancement module works as follows: according to different categories, different spectral feature channels are paid attention to. The present invention utilizes the differences between different categories (inter-class context) to enhance multi-spectral features and reduce the search space of other categories. The module is mainly divided into two parts: (1) Calculating the spectral channel attention score A. Convert the category-spectrum relationship matrix M into the channel attention score of each category; (2) Reassign weights to the spectral feature channels. Multiply the pixels of the same category with the corresponding channel attention scores, and use different channel attention scores for pixels in different category areas, thereby reducing the impact of noise in multi-spectral features and increasing inter-class differences.
[0069] See also Figure 4 , the steps of calculating the spectral channel attention score A are as follows: In this embodiment, the category-spectrum relationship matrix M is regarded as N independent spectral features, and based on the channel attention mechanism, N channel attention scores are calculated respectively to form the spectral channel attention score A. Specifically, the channel attention mechanism adopted by the present invention is a channel expansion and squeezing operation, and the squeezing rate is 32.
[0070] See also Figure 4 The steps to redistribute the weights of the spectral feature channels are as follows: The multi-spectral feature X of the lower layer is reconstructed through a convolution layer with a convolution kernel size of 1x1. l Extract features and get feature X v Next, for X p The pixel features in the corresponding area of category k are redistributed with channel weights and combined with the original multispectral features X l Spliced together, we get the output feature Y.
[0071] As an optional implementation, the decoder works as follows: the channel-enhanced spectral feature Y is input into the decoder, and is concatenated with the original features of the decoder to output a semantic segmentation result. This embodiment enables the model to output accurate segmentation results by fusing multi-spectral features with rich details and high-level features with rich semantics.
[0072] S3. Using the multispectral dataset to train the semantic segmentation model.
[0073] In the divided training set, the loss is calculated by the cross entropy function, and the parameters of the network are updated until convergence using the stochastic gradient descent algorithm. Evaluation and testing are performed on the validation set and test set respectively.
[0074] S4. Obtain an image to be processed, input the image to be processed into the trained semantic segmentation model, and output a semantic segmentation result.
[0075] The semantic segmentation method based on multispectral images in high-altitude scenes proposed in this embodiment can, on the one hand, enhance the semantic expression of pixel features within a category and reduce intra-category differences, and on the other hand, reduce the influence of noise in multispectral features and increase inter-category differences. Tables 1 and 2 show the comparison results with the best existing methods on the Potsdam dataset and the Vaihingen dataset, respectively. After applying this solution, the performance of the semantic segmentation model can be improved on both commonly used multispectral datasets.
[0076] Table 1 Comparison results of the method in this embodiment and the best existing method on the Potsdam dataset
[0077]
[0078] Table 2 Comparison results between the method of this embodiment and the best existing method on the Vaihingen dataset
[0079]
[0080] In summary, compared with the prior art, this embodiment has the following advantages and beneficial effects:
[0081] (1) The present invention uses a supervised approach to obtain a rough segmentation result, which prompts the model to correctly divide pixel features. On the other hand, it considers similar pixels with large distances or appearance differences, enhances the semantic expression of pixel features within a category, and reduces intra-category differences. Finally, the present invention extracts multispectral features for each category, that is, constructs the relationship between category and spectrum. It can effectively solve the problem of large intra-category differences.
[0082] (2) The present invention uses a channel attention mechanism to reduce redundant information of multispectral features in different categories. Since different categories focus on different spectral feature channels, the present invention calculates channel attention scores of different categories, uses different channel attention scores for pixels in different category areas, and reduces the influence of noise in multispectral features according to different categories, thereby increasing inter-class differences. It can solve the problem of interference and noise caused by redundancy and small inter-class differences.
[0083] This embodiment also provides a semantic segmentation device based on multispectral images, including:
[0084] at least one processor;
[0085] at least one memory for storing at least one program;
[0086] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.
[0087] A semantic segmentation device based on multispectral images in this embodiment can execute a semantic segmentation method based on multispectral images provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0088] The present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Figure 1 The method shown.
[0089] This embodiment also provides a storage medium, which stores instructions or programs that can execute a semantic segmentation method based on multispectral images provided by an embodiment of the method of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.
[0090] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0091] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0092] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0093] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0094] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0095] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0096] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0097] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0098] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A semantic segmentation method based on multispectral images, characterized in that: The following steps are involved: Collect and annotate multispectral datasets for semantic segmentation; Build a semantic segmentation model; Using the multispectral dataset to train the semantic segmentation model; Acquire an image to be processed, input the image to be processed into the trained semantic segmentation model, and output a semantic segmentation result; The semantic segmentation model includes a category-spectrum association module, which is used to obtain continuous and accurate segmentation results by increasing the similarity between pixels of the same category and reducing the difference between categories; The semantic segmentation model also includes an encoder, a spectral channel enhancement module and a decoder; The encoder is used to extract features from the multispectral image; The category-spectrum association module is used to obtain a preliminary segmentation result in a supervised manner, and to perform soft category mean pooling on multi-spectral features to obtain a category-spectrum relationship matrix, thereby reducing intra-category differences; The spectral channel enhancement module is used to calculate the channel attention score of each category, assign weights to the feature channels of different category areas, reduce the redundant information of different categories in multi-spectral features, and thus increase the difference between categories; The decoder is used to decode the multispectral features and output the semantic segmentation results; Define X h is the high-level feature after the first upsampling in the decoder; Define X l The low-level features output by the first stage of the backbone network in the encoder; The workflow of the category-spectrum association module is as follows: The high-level feature X h The number of channels is reduced to the number of categories N, and a softmax operation is performed on the category dimension to obtain N attention maps as the category attention map X p ; Reduce the low-level features X l The number of channels is , and feature X is obtained; Attention map X for the category p The pixel features belonging to different category areas in are respectively average-pooled to obtain N multispectral features, and the category-spectrum relationship matrix M is formed by combining the features X; The workflow of the spectral channel enhancement module is as follows: The category-spectrum relationship matrix M is regarded as N independent spectral features, and based on the channel attention mechanism, N channel attention scores are calculated respectively to form a spectral channel attention score A; For the low-level feature X l Perform feature extraction and obtain feature X v ; According to the spectral channel attention score A and the feature X v , for the category attention map X p The pixel features in the corresponding area of category k are redistributed with channel weights and combined with the low-level features X l Spliced together, we get the output feature Y.
2. The semantic segmentation method based on multispectral images according to claim 1, characterized in that: The multispectral dataset collected and annotated for semantic segmentation includes: Collecting multispectral images from high-altitude scenes and annotating the multispectral images to construct a multispectral dataset; The annotated multispectral dataset is divided into three parts: training set, validation set, and test set; For the high-resolution images in the training set, fixed-size images are randomly captured, and for the high-resolution images in the validation set and the test set, fixed-size images are captured using a sliding window.
3. The semantic segmentation method based on multispectral images according to claim 1, characterized in that: The encoder includes a backbone network and an ASPP module; The backbone network adopts dilated convolution instead of downsampling operation to increase the receptive field while preventing the resolution from decreasing. The ASPP module is used to fuse features extracted by multiple convolutional layers with different expansion rates to extract multi-scale context features.
4. The semantic segmentation method based on multispectral images according to claim 1, characterized in that: The adopting the multispectral dataset to train the semantic segmentation model comprises: The cross entropy function is used to calculate the loss, and the stochastic gradient descent algorithm is used to update the parameters of the network until convergence.
5. A semantic segmentation device based on multispectral images, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 4.
6. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 4 when executed by the processor.