Object segmentation method and related device based on residual module and attention mechanism
By combining the residual module and attention mechanism in the U-Net neural network, the problem of low ground object segmentation accuracy in remote sensing images is solved, and higher recognition accuracy and adaptability are achieved.
Patent Information
- Application Number
- CN202410326764.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-21
- Publication Date
- 2025-02-28
- Estimated Expiration
- 2044-03-21
Smart Images
Figure CN118196415B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of remote sensing object segmentation, and in particular to a method for object segmentation based on a residual module and an attention mechanism and related devices. Background Art
[0002] In recent years, remote sensing technology and computer technology have developed rapidly. Many researchers are committed to using machine learning algorithms to improve the efficiency of image processing. As the performance of computer hardware continues to improve, deep learning technology, which requires powerful computing power to support, has developed rapidly in the field of computer vision and has shown great potential in monitoring and identification work such as remote sensing, autonomous driving, and medical image processing.
[0003] Deep convolutional neural networks are the representative of deep learning algorithms for image processing. In recent years, neural network models for various problems have been rapidly derived. Among them, fully convolutional neural networks (FCN) and U-Net models are widely used in the field of image segmentation and are the mainstream networks currently used. U-Net was first used in medical image segmentation tasks and is known for its simple structure and excellent performance. Therefore, different improvements have been made according to different problems, such as UNet++, Attention U-Net, U2-Net and other excellent models.
[0004] Introducing deep learning-based image semantic segmentation technology into the field of remote sensing and using computers to automatically interpret remote sensing is a necessary path and a huge challenge. However, the information of objects in remote sensing images is complex and easily confused with each other. There are problems such as unclear boundaries between objects and low contrast. Simple convolution operations can extract global features of the image, but the lack of spatial correlation information will increase the weight of redundant pixels and affect the recognition accuracy of the target land type. Summary of the invention
[0005] The purpose of the present invention is to provide a method and related devices for object segmentation based on a residual module and an attention mechanism, so as to improve the accuracy of object segmentation in remote sensing images.
[0006] To achieve the above object, the present invention provides the following solutions:
[0007] On the one hand, the present invention provides a method for object segmentation based on a residual module and an attention mechanism, comprising the following steps:
[0008] Acquire a remote sensing image to be segmented; the remote sensing image to be segmented includes several buildings and ground areas.
[0009] The remote sensing image to be segmented is input into the trained object segmentation model to obtain the object segmentation result; the object segmentation result includes the building segmentation result; the object segmentation model is a network model based on the U-Net neural network, combined with the residual module and the attention module.
[0010] Optionally, the method further comprises the following steps:
[0011] A training data set is obtained; the training data set includes a number of training samples; the training samples include training remote sensing images and corresponding building segmentation labels.
[0012] Based on the U-Net neural network, a ground object segmentation model is constructed by combining the residual module and the attention module.
[0013] The ground object segmentation model is trained using the training data set to obtain a trained building segmentation model.
[0014] Optionally, the ground object segmentation model is trained using the training data set to obtain a trained building segmentation model, specifically including:
[0015] For any training sample, the training remote sensing image of the training sample is input into the object segmentation model, and the building segmentation label of the training sample is used as the target output to update the model parameters of the object segmentation model.
[0016] Optionally, the method further comprises the following steps:
[0017] A test data set is obtained; the test data set includes several test samples; the test samples include a test remote sensing image and corresponding building segmentation labels.
[0018] The trained land object segmentation model is tested using the test data set to determine the building segmentation accuracy of the trained land object segmentation model.
[0019] Optionally, the object segmentation model includes an encoder and a decoder; the encoder and the decoder include a residual module; and the decoder includes an attention module.
[0020] Optionally, the encoder includes a first convolution block, a second convolution block, a third convolution block, a fourth convolution block and a fifth convolution block connected in sequence; the residual module includes a first residual block, a second residual block, a third residual block and a fourth residual block; the first residual block is connected between the first convolution block and the second convolution block; the second residual block is connected between the second convolution block and the third convolution block; the third residual block is connected between the third convolution block and the fourth convolution block; and the fourth residual block is connected between the fourth convolution block and the fifth convolution block.
[0021] Optionally, the decoder includes a sixth convolution block, a seventh convolution block, an eighth convolution block and a ninth convolution block connected in sequence; the attention module includes: a mixed domain attention block, a first cross attention block, a second cross attention block, a third cross attention block and a fourth cross attention block; the residual module also includes a fifth residual block, a sixth residual block, a seventh residual block and an eighth residual block.
[0022] The fifth convolution module is connected to the sixth convolution block through the mixed domain attention block; the sixth convolution block is connected to the seventh convolution module through the fifth residual block; the seventh convolution block is connected to the eighth convolution module through the sixth residual block; the eighth convolution block is connected to the ninth convolution module through the seventh residual block; the output of the ninth convolution block is output through the eighth residual block.
[0023] The first cross-attention block splices the output of the fourth residual block to the output of the sixth convolution block; the second cross-attention block splices the output of the third residual block to the output of the seventh convolution block; the third cross-attention block splices the output of the second residual block to the output of the eighth convolution block; the fourth cross-attention block splices the output of the first residual block to the output of the ninth convolution block.
[0024] On the other hand, the present invention provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-mentioned methods for object segmentation based on a residual module and an attention mechanism.
[0025] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods for object segmentation based on a residual module and an attention mechanism.
[0026] On the other hand, the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for object segmentation based on a residual module and an attention mechanism.
[0027] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0028] The present invention provides a method and related device for object segmentation based on a residual module and an attention mechanism, the method comprising the following steps: obtaining a remote sensing image to be segmented; inputting the remote sensing image to be segmented into a trained object segmentation model to obtain an object segmentation result; wherein the object segmentation model is a network model obtained by combining a residual module and an attention module based on a U-Net neural network. The present invention adopts a U-Net model that combines a residual network structure and an attention mechanism. The main idea is to use an improved residual structure to avoid the degradation problem of a deep network model. In addition, a hybrid attention mechanism and a cross-attention mechanism are introduced, so that the model has the ability to connect to long-distance contextual information, can make more full use of the global information of the image, and enhance the network's adaptive ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0030] Figure 1 A flowchart of a method for segmenting objects based on a residual module and an attention mechanism provided in Example 1 of the present invention;
[0031] Figure 2 A flowchart of steps B1 to B3 in a method for segmenting objects based on a residual module and an attention mechanism provided in Example 1 of the present invention;
[0032] Figure 3 A schematic diagram of an aerial imagery dataset building data set in a method for segmenting objects based on a residual module and an attention mechanism provided in Example 1 of the present invention;
[0033] Figure 4 A schematic diagram of a Satellite dataset II building dataset in a method for segmenting objects based on a residual module and an attention mechanism provided in Example 1 of the present invention;
[0034] Figure 5 A schematic diagram of the structure of a ground object segmentation model in a ground object segmentation method based on a residual module and an attention mechanism provided in Example 1 of the present invention;
[0035] Figure 6 A schematic diagram of the structure of a residual convolution module in a method for object segmentation based on a residual module and an attention mechanism provided in Example 1 of the present invention;
[0036] Figure 7A schematic diagram of the structure of a mixed domain attention block in a method for object segmentation based on a residual module and an attention mechanism provided in Example 1 of the present invention;
[0037] Figure 8 This is a diagram of the internal structure of a computer device provided in Example 4 of the present invention. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] The purpose of the present invention is to provide a method and related device for object segmentation based on a residual module and an attention mechanism, aiming to improve the accuracy of object segmentation in remote sensing images.
[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0041] Example 1
[0042] like Figure 1 As shown in the flowchart, a method for segmenting objects based on a residual module and an attention mechanism in this embodiment includes the following steps:
[0043] A1. Obtain the remote sensing image to be segmented. The remote sensing image to be segmented includes several buildings and ground areas.
[0044] A2. Input the remote sensing image to be segmented into the trained object segmentation model to obtain the object segmentation result. The object segmentation result includes the building segmentation result; the object segmentation model is a network model based on the U-Net neural network, combined with the residual module and the attention module.
[0045] Specifically, before using the object segmentation model, it is necessary to build and train the object segmentation model, such as Figure 2 The flowchart shown includes the following steps:
[0046] B1. Obtain a training data set. The training data set includes several training samples; the training samples include training remote sensing images and corresponding building segmentation labels.
[0047] In this embodiment, the training data set used for model training comes from the WHU building semantic segmentation aerial imagery dataset. Using the WHU building semantic segmentation dataset, 5000 image samples are randomly selected, with an image size of 512×512 (pixels) and a ground resolution of 0.3m. Figure 3 As shown, the white part is the building, the black part is the background, and 80% of the samples are randomly selected as the training data set and 20% as the test data set.
[0048] In another embodiment, the training dataset and the test dataset are constructed using the Satellite dataset II in the WHU building semantic segmentation dataset. 5000 image samples are randomly selected, the image size is 512×512 (pixels), and the ground resolution is 0.45m. Figure 4 As shown, the white part is the building, the black part is the background, and 80% of the samples are randomly selected as the training data set and 20% as the test data set.
[0049] B2. Based on the U-Net neural network, the residual module and the attention module are combined to build a ground object segmentation model. Figure 5 As shown in the schematic diagram of the network model structure, the object segmentation model includes an encoder and a decoder; the encoder and the decoder include a residual module; and the decoder includes an attention module.
[0050] The encoder includes a first convolution block, a second convolution block, a third convolution block, a fourth convolution block and a fifth convolution block connected in sequence; the residual module includes a first residual block, a second residual block, a third residual block and a fourth residual block; the first residual block is connected between the first convolution block and the second convolution block; the second residual block is connected between the second convolution block and the third convolution block; the third residual block is connected between the third convolution block and the fourth convolution block; and the fourth residual block is connected between the fourth convolution block and the fifth convolution block.
[0051] Each residual convolutional module (adjacent convolutional blocks and residual blocks) in the encoder is followed by an average pooling layer. Figure 6 The structure of the residual convolution module shown in the figure replaces the second activation function in the residual network with the Mish function instead of the ReLU function, which has better generalization and more effective optimization capabilities. The purpose of adding residual block connections to the convolution layer is to retain more of the original information of the input image and pass it to the deeper layers of the network. When the number of layers of the convolutional neural network is very deep, the residual connection can solve the problems of gradient disappearance and gradient explosion, and can also solve the problem of performance degradation when the number of layers of the network is deepened.
[0052] The decoder includes a sixth convolution block, a seventh convolution block, an eighth convolution block and a ninth convolution block connected in sequence; the attention module includes: a mixed domain attention block, a first cross attention block, a second cross attention block, a third cross attention block and a fourth cross attention block; the residual module also includes a fifth residual block, a sixth residual block, a seventh residual block and an eighth residual block.
[0053] like Figure 7 The structure shown in the figure shows that the mixed domain attention block mainly includes two types of attention, namely the spatial attention block and the channel attention block. The spatial attention block selectively aggregates each feature by assigning weights to the features at each position, and all similar features will be related to each other. The channel attention block selectively emphasizes the channel mappings that are interdependent by integrating the relevant features between all channel mappings.
[0054] The fifth convolution module is connected to the sixth convolution block through the mixed domain attention block; the sixth convolution block is connected to the seventh convolution module through the fifth residual block; the seventh convolution block is connected to the eighth convolution module through the sixth residual block; the eighth convolution block is connected to the ninth convolution module through the seventh residual block; the output of the ninth convolution block is output through the eighth residual block.
[0055] The first cross-attention block splices the output of the fourth residual block to the output of the sixth convolution block; the second cross-attention block splices the output of the third residual block to the output of the seventh convolution block; the third cross-attention block splices the output of the second residual block to the output of the eighth convolution block; the fourth cross-attention block splices the output of the first residual block to the output of the ninth convolution block.
[0056] B3, using the training data set, training the ground feature segmentation model to obtain a trained building segmentation model. Specifically, step B3 includes the following steps:
[0057] For any training sample, the training remote sensing image of the training sample is input into the object segmentation model, and the building segmentation label of the training sample is used as the target output to update the model parameters of the object segmentation model.
[0058] The data set obtained in step B1 is substituted into the improved UNet model for training. During the model training process, the original image of the data set obtained in step B1 is input, and the building segmentation image is output.
[0059] In order to solve the problem of overfitting in training due to small sample size, the learning rate decay method is used to dynamically adjust the learning rate to prevent overfitting and ensure a certain learning rate. The optimizer uses Adam, the loss function uses the binary cross entropy loss function (BCE Loss), the number of iterations (Epoch) is set to 50, the number of batches (Batch Size) is set to 2, and the initial learning rate is set to 0.001. The loss function BCE Loss can be calculated according to the following formula:
[0060] L=-(ylog(p)+(1-y)log(1-p)).
[0061] Among them, L is the value of the cross entropy loss function, y is the true label (0 or 1), and p is the probability of the model predicting the positive class.
[0062] The computer hardware environment of the experiment is an i7-5930K processor, a Tesla-V100 graphics card with 64GB of RAM and 32GB of video memory. The software environment is a 64-bit Windows 10 operating system and a Pytorch deep learning framework.
[0063] After training the object segmentation model, the following steps are also included:
[0064] C1. Obtain a test data set; the test data set includes several test samples; the test samples include a test remote sensing image and corresponding building segmentation labels.
[0065] C2. Use the test data set to test the trained land object segmentation model to determine the building segmentation accuracy of the trained land object segmentation model.
[0066] The superiority of the solution of the present invention is demonstrated by comparing the segmentation and recognition accuracy of the traditional U-Net network model, the ResU-Net network model with only the residual module added, and the ResU-Net+Attention network model with the residual module and the attention module added. The comparison results are shown in Table 1. According to Table 1, the overall segmentation accuracy of the ResU-Net network model is improved from 90.61% of U-Net to 92.35%, and the F1 value is also improved from 86.13% to 87.07%. It can be seen that the residual module can effectively improve the segmentation accuracy. The attention module in ResU-Net+Attention can distinguish similar objects more accurately, further improving the overall accuracy of the model from 92.35% to 94.33%, and the F1 value is also improved from 87.07% to 88.93%.
[0067] Table 1 Comparison of segmentation accuracy of different improved models
[0068]
[0069] The improved ResU-Net+Attention convolutional network model proposed in this invention is applied to the Satellite datasetⅡ and Aerial imagery dataset of the WHU building dataset, and the segmentation accuracy comparison results on different datasets are shown in Table 2. The average F1 value of the improved model is 88.48%, and the average overall accuracy is 93.82%. It can be concluded that the model has good reliability and accuracy for the segmentation of single objects in remote sensing images.
[0070] Table 2 Comparison of segmentation accuracy on different datasets
[0071]
[0072] This embodiment provides a method for object segmentation based on a residual module and an attention mechanism, obtains a remote sensing image to be segmented; inputs the remote sensing image to be segmented into a trained object segmentation model to obtain an object segmentation result; wherein the object segmentation model is a network model obtained by combining a residual module and an attention module based on a U-Net neural network. This embodiment adopts a U-Net model that combines a residual network structure and an attention mechanism. The main idea is to use an improved residual structure to avoid the degradation problem of a deep network model. In addition, a hybrid attention mechanism and a cross-attention mechanism are introduced, so that the model has the ability to connect to long-distance contextual information, can make more full use of the global information of the image, and enhance the network's adaptive ability.
[0073] Example 2
[0074] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method for segmenting objects based on a residual module and an attention mechanism in Example 1.
[0075] Example 3
[0076] A computer program product includes a computer program, which, when executed by a processor, implements the steps of a method for segmenting objects based on a residual module and an attention mechanism in Example 1.
[0077] Example 4
[0078] A computer device, which may be a database, may have an internal structure as shown in Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for segmenting objects based on a residual module and an attention mechanism in Example 1 is implemented.
[0079] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0080] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present invention may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.
[0081] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0082] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A method for object segmentation based on residual module and attention mechanism, characterized in that: include: Obtaining remote sensing images to be segmented; The remote sensing image to be segmented includes a number of buildings and ground areas; Inputting the remote sensing image to be segmented into a trained object segmentation model to obtain an object segmentation result; the object segmentation result includes a building segmentation result; the object segmentation model is a network model based on a U-Net neural network, combined with a residual module and an attention module; The object segmentation model includes an encoder and a decoder; the encoder and the decoder include a residual module, and the decoder includes an attention module; The encoder includes a first convolution block, a second convolution block, a third convolution block, a fourth convolution block and a fifth convolution block connected in sequence; the decoder includes a sixth convolution block, a seventh convolution block, an eighth convolution block and a ninth convolution block connected in sequence; the attention module includes: a mixed domain attention block, a first cross attention block, a second cross attention block, a third cross attention block and a fourth cross attention block; The mixed domain attention block mainly includes two types of attention, namely, a spatial attention block and a channel attention block; the fifth convolution block is connected to the sixth convolution block through the mixed domain attention block; Each residual convolution module in the encoder is followed by an average pooling layer, and the residual convolution module refers to adjacent convolution layers and residual blocks in the encoder; in the residual convolution module, the second activation function of the residual network is the Mish function.
2. According to claim 1, a method for segmenting objects based on a residual module and an attention mechanism is characterized in that: Also includes: Acquire a training data set; the training data set includes a plurality of training samples; the training samples include training remote sensing images and corresponding building segmentation labels; Based on the U-Net neural network, the residual module and the attention module are combined to build a ground object segmentation model; The ground object segmentation model is trained using the training data set to obtain a trained building segmentation model.
3. According to claim 2, a method for segmenting objects based on a residual module and an attention mechanism is characterized in that: Using the training data set, the ground feature segmentation model is trained to obtain a trained building segmentation model, specifically including: For any training sample, the training remote sensing image of the training sample is input into the land object segmentation model, the building segmentation label of the training sample is used as the target output, and the model parameters of the land object segmentation model are updated.
4. The method for segmenting objects based on residual module and attention mechanism according to claim 1, characterized in that: Also includes: Acquire a test data set; the test data set includes a plurality of test samples; The test sample includes a test remote sensing image and a corresponding building segmentation label; The trained land object segmentation model is tested using the test data set to determine the building segmentation accuracy of the trained land object segmentation model.
5. The method for object segmentation based on residual module and attention mechanism according to claim 1, characterized in that: The residual module includes a first residual block, a second residual block, a third residual block and a fourth residual block; the first residual block is connected between the first convolution block and the second convolution block; the second residual block is connected between the second convolution block and the third convolution block; the third residual block is connected between the third convolution block and the fourth convolution block; the fourth residual block is connected between the fourth convolution block and the fifth convolution block.
6. The method for segmenting objects based on residual module and attention mechanism according to claim 5, characterized in that: The residual module further includes a fifth residual block, a sixth residual block, a seventh residual block and an eighth residual block; The sixth convolution block is connected to the seventh convolution module through the fifth residual block; the seventh convolution block is connected to the eighth convolution module through the sixth residual block; the eighth convolution block is connected to the ninth convolution module through the seventh residual block; the output of the ninth convolution block is output through the eighth residual block; The first cross-attention block splices the output of the fourth residual block to the output of the sixth convolution block; the second cross-attention block splices the output of the third residual block to the output of the seventh convolution block; the third cross-attention block splices the output of the second residual block to the output of the eighth convolution block; the fourth cross-attention block splices the output of the first residual block to the output of the ninth convolution block.
7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of a method for object segmentation based on a residual module and an attention mechanism as described in any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for segmenting objects based on a residual module and an attention mechanism described in any one of claims 1 to 6 are implemented.
9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of a method for segmenting objects based on a residual module and an attention mechanism described in any one of claims 1 to 6 are implemented.