An image segmentation method, system and electronic device based on multi-layer feature concatenation

Abstract

The application relates to an image segmentation method, system and electronic equipment based on a multi-layer feature cascade, which is realized based on an improved DeepLabv3+ model and comprises the following steps: inputting an original image of a water surface unmanned ship obtained to a feature extraction module, outputting low-layer feature maps after multiple times of downsampling and a feature map x1 passing through the entire feature extraction module; inputting the output feature map x1 to a coordinate attention module CA to obtain spatial position information; inputting the output result of the coordinate attention module CA in the step to an improved CASPP to obtain a high-layer feature map; fusing and splicing the low-layer feature maps and the high-layer feature map of different scales, and outputting an image segmentation result. Compared with the prior art, the application has the advantages of realizing accurate image segmentation, improving feature extraction efficiency and providing efficient guidance for navigation decision, etc.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image segmentation method, system, and electronic device based on multi-layer feature cascading. Background Technology

[0002] With the advancement of science and technology, image processing technology is increasingly being applied to daily life and production. Many technologies incorporate image processing techniques, such as autonomous driving, self-checkout machines, and beauty cameras. Image processing technology can be further subdivided into image enhancement, image segmentation, image recognition, and image restoration. Image segmentation also has wide applications in life, such as medical image analysis and unmanned surface vessel navigation. In the medical field, image segmentation is used to analyze medical images such as X-rays, CT scans, and MRIs to help doctors identify and diagnose diseases. For example, segmented images can be used to measure the size, location, and shape of tumors, or to help analyze bone structure. For unmanned surface vessels, the autonomous driving system uses image segmentation to understand the surrounding environment. By segmenting the water surface, sky, small obstacles, and other vessels, the system can better understand different elements in the water scene and make corresponding navigation decisions. There are several typical segmentation models in image segmentation, with FCN, U-Net, SegNet, and DeepLab all showing good performance. The latest model in the DeepLab series is DeepLabv3+, which adds a simple yet effective decoder module to the basic DeepLabv3 architecture for better recovery of spatial information and finer segmentation boundaries. Its ASPP module also allows the network to process receptive fields of different sizes simultaneously, improving the accuracy of semantic segmentation. However, the DeepLabv3+ model also has limitations. It is relatively limited in its handling of details, and may not be able to process all the detailed information for very small objects and edges. DeepLabv3+ also has high computational complexity, is difficult to train, and may have a slower inference speed.

[0003] How to achieve accurate and rapid segmentation of images from unmanned surface vessels has become a technical problem that needs to be solved. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an image segmentation method, system, and electronic device based on multi-layer feature cascading, thereby achieving better feature extraction, obtaining contextual information, and improving segmentation accuracy.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] According to one aspect of the present invention, an image segmentation method based on multi-layer feature concatenation is provided. This method is implemented based on an improved DeepLabv3+ model and includes the following steps:

[0007] Step S1: Input the acquired original image of the unmanned surface vessel into the feature extraction module, and output the low-level feature map after multiple downsampling and the feature map x1 that passes through the entire feature extraction module;

[0008] Step S2: Input the feature map x1 output in step S1 into the coordinate attention module CA to obtain spatial location information;

[0009] Step S3: Input the result of step S2 into the improved CASPP to obtain the high-level feature map;

[0010] Step S4: The low-level feature maps of different scales from step S1 are fused and stitched together with the high-level feature maps, and the image segmentation result is output.

[0011] Preferably, the low-level feature map includes: a 1 / 8 downsampled feature map (down). 1 / 8 (x) and the feature map down sampled by 1 / 4 1 / 4 (x).

[0012] More preferably, the fusion and stitching of low-level feature maps and high-level feature maps of different scales in step S1 includes:

[0013] Step S101: Downsample the feature map by 1 / 8. 1 / 8 (x) is divided into two parts, and one part is downsampled by 1 / 8 of the feature map. 1 / 8 (x), after adjusting the channel, it is spliced ​​with the result of upsampling the high-level feature map of step S3 by twice;

[0014] Step S102, take another 1 / 8 downsampled feature map down 1 / 8 (x) is first upsampled by a factor of two, then passed through a 3×3 convolutional layer with an inflation rate of 6, and finally downsampled with a feature map of 1 / 4. 1 / 4 (x) is used for splicing;

[0015] Step S103: The result of step S102 is combined with the result of upsampling by twice in step S101.

[0016] More preferably, the output image segmentation result includes: adjusting the number of channels of the result in step S103 and then performing upsampling by four times to restore the image size, and outputting the image segmentation result.

[0017] Preferably, the improved CASPP adopts a multi-layer structure fusion approach, replacing the original convolution with two separate convolutions.

[0018] More preferably, the improved CASPP calculation formula is as follows:

[0019]

[0020]

[0021] x 3_5 =f pooling (x2)

[0022] x3=f Concat (x 3_ ,x 3_ ,x 3_ ,x 3_4 ,x 3_5 )

[0023] Where x2 represents the input to CASPP, f represents a 1×1 convolution operation. pooling This indicates a global average pooling operation. This represents a 3×1 dilated convolution with an inflation rate of 6. This represents a 1×3 dilated convolution with an inflation rate of 6. This represents a 3×1 dilated convolution with an inflation rate of 12. This represents a 3×1 dilated convolution with an inflation rate of 18. This represents a 1×3 dilated convolution with an inflation rate of 18, f Concat This indicates a splicing operation.

[0024] Preferably, the coordinate attention module (CA) can capture not only cross-channel information, but also direction awareness and position awareness information.

[0025] According to another aspect of the present invention, an image segmentation system based on multi-layer feature cascading is provided. The system includes an image acquisition and preprocessing module and a segmentation module, wherein the segmentation module includes an encoder and a decoder. The image acquisition and preprocessing module is used to preprocess the acquired raw images of unmanned surface vessels and then input them into the encoder. The encoder is used to extract features from the input images and includes a feature extraction module, a coordinate attention module (CA), and a CASPP module. The decoder is used to fuse and stitch together the high-level feature maps output by the encoder and low-level feature maps of different scales, and output the image segmentation result.

[0026] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method.

[0027] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1) This invention introduces a coordinate attention module (CA), which not only captures cross-channel information but also captures direction-aware and position-aware information, preserving spatial location information and enabling the segmentation model to more accurately locate and identify targets of interest. A cascading operation is added to the original ASPP module, allowing feature maps of different scales to exchange information and acquire more detailed information from the feature maps. The fusion of low-level and high-level features at different scales effectively solves the problem of boundary information loss caused by multiple downsampling, effectively improving the model's performance on semantic segmentation tasks, thereby achieving accurate image segmentation and providing effective guidance for navigation decisions.

[0030] 2) The improved ASPP module replaces the original 3×3 convolution with split convolution, turning it into two convolutions: 3×1 and 1×3. This effectively reduces the number of parameters, lowers the training difficulty and inference speed of the model, thereby improving feature extraction efficiency and enhancing the real-time performance of navigation decisions. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the structure of the model in this invention;

[0032] Figure 2 This is a schematic diagram of the improved CASPP structure in this invention;

[0033] Figure 3 This is a flowchart illustrating the image segmentation method of the present invention;

[0034] In the attached diagram, Image: image, Encoder: encoder, Decoder: decoder, Upsample: upsampling, Conv: convolution, Prediction: segmentation prediction, Concat: concatenation, pooling: pooling, CA: coordinate attention module. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] This embodiment relates to an image segmentation method based on multi-layer feature cascading, which includes the following steps:

[0037] Step S1: Input the original image of the unmanned surface vessel into the Xception feature extraction module for feature extraction.

[0038] Step S2: Input the feature map extracted in step S1 into the coordinate attention module CA.

[0039] Step S3: Input the results from step S2 into the improved CASPP.

[0040] Step S4: Take the feature map that was downsampled by 1 / 8 in step S1, adjust the channels, and then stitch it together with the result that was upsampled by 2 times in step S3.

[0041] Step S5: Take the feature map that was downsampled by 1 / 8 in step S1, first upsample it by 2 times, then pass it through a 3×3 convolutional layer with an inflation rate of 6, and then concatenate it with the feature map that was downsampled by 1 / 4 in step S1.

[0042] Step S6: Combine the result from step S5 with the result from step S4 that was upsampled by 2.

[0043] Step S7: Adjust the number of channels of the result from step S6 and perform upsampling by 4 times to restore the image size and output the image segmentation result.

[0044] Step S1 includes: transferring the original image The input is processed by the feature extraction network Xception, which performs multiple downsampling operations to obtain 1 / 4 of the downsampled data. 1 / 4 (x) and 1 / 8 downsampling 1 / 8(x) and the result x1 after passing through the entire network. The entire feature extraction network is divided into 3 parts: Entry flow, Middleflow, and Exit flow, with a total of 24 modules. Except for the first and last modules, each module has residual structure connections. All max pooling is replaced by depthwise separable convolutions with stride=2; each 3×3 depthwise convolution is followed by batch normalization (BN) and ReLU activation function.

[0045] Step S2 includes: passing the results obtained in step S1 through the coordinate attention module CA. Coordinate attention can not only capture cross-channel information, but also capture orientation and position information and retain spatial position information, which can help the model to more accurately locate and identify targets of interest.

[0046] x2 = CA(x1)

[0047] Where x1 is the result of the original image x passing through the entire feature extraction network.

[0048] Step S3 includes: inputting the result x2 obtained in step S2 into the improved CASPP module of the present invention, which can enhance the fusion of spatial context information.

[0049] The CASPP module of this invention improves upon the original simple parallel structure of ASPP by employing a multi-layered fusion approach. This results in a closer connection between contextual information, allowing for the extraction of more detailed information from the feature maps. Furthermore, the original 3×3 convolution is replaced with a split convolution, resulting in two convolutions: a 3×1 convolution and a 1×3 convolution. This effectively reduces the number of parameters and improves feature extraction capabilities. The calculation formula is as follows:

[0050]

[0051] x 3_5 =f pooling (x2)

[0052] x3=f Concat (x 3_ ,x 3_ ,x 3_ ,x 3_4 ,x 3_5 )

[0053] in, f represents a 1×1 convolution operation. pooling This indicates a global average pooling operation. This represents a 3×1 dilated convolution with an inflation rate of 6. This represents a 1×3 dilated convolution with an inflation rate of 6. This represents a 3×1 dilated convolution with an inflation rate of 12. This represents a 3×1 dilated convolution with an inflation rate of 18. This represents a 1×3 dilated convolution with an inflation rate of 18, f Concat This indicates a splicing operation.

[0054] Step S4 includes: adjusting the number of channels of the 1 / 8 downsampled feature map obtained from the Xception feature extraction network in step S1 through a 1×1 convolution to reduce computation; adjusting the number of channels of the result from step S3 through a 1×1 convolution, and then upsampling it by a factor of 2, so that the size of the feature map after CASPP is similar to the downsampled feature map. 1 / 8 (x) have the same size, and then the two results are spliced ​​and merged.

[0055]

[0056] Where Up2 represents upsampling by 2, f Concat_Conv This indicates that convolution, normalization, and activation are performed after concatenation. This represents a 1×1 convolution operation, down. 1 / 8 (x) indicates 1 / 8 downsampling.

[0057] Step S5 includes: obtaining 1 / 8 and 1 / 4 downsampled feature maps from the Xception feature extraction network in step S1; upsampling the 1 / 8 downsampled feature map by a factor of 2 to make it consistent with the size of the 1 / 4 downsampled feature map; then passing it through a 3×3 convolutional layer with an inflation rate of 6 to obtain a larger receptive field; and finally, after normalization, concatenating and fusing it with the 1 / 4 downsampled feature map.

[0058]

[0059] Where BN represents batch normalization. This represents a 1×1 convolution operation, down. 1 / 8 (x) represents 1 / 8 downsampling, down 1 / 4 (x) represents 1 / 4 downsampling, and Up2 represents upsampling by 2 times. f represents a 3×3 dilated convolution with an inflation rate of 6. Concat This indicates a splicing operation.

[0060] Step S6 includes: upsampling the result of step S4 by 2 times, keeping the size consistent with the result of step S5, and then stitching and merging it with the result of step S5.

[0061] x6=f Concat_Conv (x5,Up2(x4))

[0062] Where BN represents batch normalization, Up2 represents upsampling by 2, and f Concat_Conv This indicates that convolution, normalization, and activation are performed after splicing.

[0063] Step S7 includes: performing a 3×3 convolution on the result of step S6, adjusting the number of channels, and outputting the number of channels equal to the number of categories. Finally, a 4x upsampling is performed to restore the spatial size of the feature map to the same size as the input image, improving the accuracy and reliability of segmentation.

[0064]

[0065] in, Up4 indicates a 3×3 convolution operation, and Up4 indicates an upsampling factor of 4.

[0066] This embodiment also relates to an image segmentation system based on multi-layer feature cascading, including an image acquisition and preprocessing module and a segmentation module, wherein the segmentation module includes an encoder and a decoder; the image acquisition and preprocessing module is used to preprocess the acquired raw images of unmanned surface vessels and then input them into the encoder; the encoder is used to extract features from the input image, including a feature extraction module, a coordinate attention module (CA), and a CASPP module; the decoder is used to fuse and stitch together the high-level feature maps output by the encoder and low-level feature maps of different scales, and output the image segmentation result.

[0067] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0068] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0069] The processing unit executes the various methods and processes described above, such as methods S1 to S7. For example, in some embodiments, methods S1 to S7 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S7 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S7 by any other suitable means (e.g., by means of firmware).

[0070] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0071] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0072] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0073] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image segmentation method based on multi-layer feature cascading, characterized in that, This method is based on an improved DeepLabv3+ model and includes the following steps: Step S1: Input the acquired raw image of the unmanned surface vessel into the feature extraction module, and output the low-level feature map after multiple downsampling and the feature map x1 that has passed through the entire feature extraction module; the low-level feature map includes: a 1 / 8 downsampled feature map down 1 / 8 (x) and the feature map down sampled by 1 / 4 1 / 4 (x); Step S2: Input the feature map x1 output in step S1 into the coordinate attention module CA to obtain spatial location information; Step S3: Input the result of step S2 into the improved CASPP to obtain the high-level feature map; The improved CASPP employs a multi-layer structure fusion approach, replacing the original convolution with two separate convolutions; the calculation formula for the improved CASPP is as follows: x 3_1 = x 3_2 = x 3_3 = x 3_4 = x 3_5 = x3= in, This represents the input to CASPP. This represents a 1×1 convolution operation. This indicates a global average pooling operation. This represents a 3×1 dilated convolution with an inflation rate of 6. This represents a 1×3 dilated convolution with an inflation rate of 6. This represents a 3×1 dilated convolution with an inflation rate of 12. This represents a 3×1 dilated convolution with an inflation rate of 18. This represents a 1×3 dilated convolution with an inflation rate of 18. Indicates a splicing operation; Step S4: Fuse and stitch the low-level feature maps of different scales from step S1 with the high-level feature maps, and output the image segmentation result. The process of fusing and stitching low-level feature maps and high-level feature maps of different scales in step S1 includes: Step S101, downsample the feature map by 1 / 8. 1 / 8 (x) is divided into two parts, and one part is downsampled by 1 / 8 of the feature map. 1 / 8 (x), after adjusting the channel, it is spliced ​​with the result of upsampling the high-level feature map of step S3 by twice; Step S102, take another 1 / 8 downsampled feature map down 1 / 8 (x) is first upsampled by a factor of two, then passed through a 3×3 convolutional layer with an inflation rate of 6, and finally downsampled with a feature map of 1 / 4. 1 / 4 (x) is used for splicing; Step S103: The result of step S102 is combined with the result of upsampling by twice in step S101.

2. The image segmentation method based on multi-layer feature cascading according to claim 1, characterized in that, The output image segmentation result includes: adjusting the number of channels of the result in step S103, performing upsampling by four times, restoring the image size, and outputting the image segmentation result.

3. The image segmentation method based on multi-layer feature cascading according to claim 1, characterized in that, The aforementioned coordinate attention module (CA) can capture not only cross-channel information, but also direction awareness and position awareness information.

4. A system employing the image segmentation method based on multi-layer feature cascading as described in claim 1, characterized in that, The system includes an image acquisition and preprocessing module and a segmentation module, wherein the segmentation module includes an encoder and a decoder; the image acquisition and preprocessing module is used to preprocess the acquired raw images of the unmanned surface vessel and then input them into the encoder; the encoder is used to extract features from the input images and includes a feature extraction module, a coordinate attention module (CA), and a CASPP module. The decoder is used to fuse and stitch together the high-level feature maps output by the encoder and the low-level feature maps of different scales, and output the image segmentation result.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 3.

Images