A feature extraction network and a feature extraction method for semantic segmentation

Abstract

The application provides a feature extraction network and a feature extraction method for semantic segmentation, the network comprising: at least two feature extraction modules and at least one ASPP module, wherein any one of the at least one ASPP module is located between any two adjacent feature extraction modules of the at least two feature extraction modules; the ASPP module is used for acquiring first features output by a previous adjacent feature extraction module, outputting second features after processing the first features, and inputting the first features and the second features into a next adjacent feature extraction module after adding the first features and the second features. By adding an ASPP module between the at least two feature extraction modules, the ASPP module is used for fusing large receptive field features and small receptive field features, so that more context information and detail information can be extracted by the semantic segmentation network at the same time, and the effect of semantic segmentation is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a feature extraction network and feature extraction method for semantic segmentation. Background Technology

[0002] Semantic segmentation deep learning networks generally include encoder-decoder structures and dual-branch parallel structures. The encoder-decoder structure consists of an encoder and a decoder. The encoder first progressively downsamples the input to extract features, and then the decoder progressively upsamples the encoder's output to obtain the semantic segmentation result. The feature extraction process of the dual-branch parallel structure includes encoder feature extraction, feature fusion, decoder deconvolution, and feature adjustment. These steps work together to extract rich and meaningful feature representations while preserving the positional and detailed information of the original image, thus achieving accurate semantic segmentation.

[0003] In related technologies, for encoding and decoding structures, downsampling is usually reduced in the encoder stage to retain more detailed information, but this can lead to insufficient extraction of contextual information. On the other hand, not reducing downsampling in the encoder stage can lead to loss of detailed information. For dual-branch parallel structures, downsampling is performed in both branches, which can also lead to loss of details. Summary of the Invention

[0004] This invention provides a feature extraction network and method for semantic segmentation, which solves the defects of loss of detailed information and inability to fully extract contextual information in the feature extraction process of existing technologies, and realizes the ability to extract more contextual information and detailed information at the same time.

[0005] In a first aspect, the present invention provides a feature extraction network for semantic segmentation, comprising:

[0006] At least two feature extraction modules and at least one ASPP module, wherein any one of the at least two ASPP modules is located between any two adjacent feature extraction modules of the at least two feature extraction modules;

[0007] The ASPP module is used to obtain the first feature output by the previous adjacent feature extraction module, process the first feature and output the second feature, and add the first feature and the second feature and input them into the next adjacent feature extraction module.

[0008] According to the present invention, a feature extraction network for semantic segmentation is provided, wherein the ASPP module is used to perform feature extraction processing and / or pooling processing on the first feature.

[0009] According to the present invention, a feature extraction network for semantic segmentation is provided, wherein the at least two feature extraction modules sequentially include: a 1 / 4 feature extraction module, a 1 / 8 feature extraction module, a 1 / 16 feature extraction module, and a 1 / 32 feature extraction module.

[0010] According to the feature extraction network for semantic segmentation provided by the present invention, the at least one ASPP module includes one or more of the following: a first ASPP module, a second ASPP module, and a third ASPP module;

[0011] The preceding adjacent feature extraction module of the first ASPP module is a 1 / 4 feature extraction module, and the following adjacent feature extraction module of the first ASPP module is a 1 / 8 feature extraction module.

[0012] The preceding adjacent feature extraction module of the second ASPP module is a 1 / 8 feature extraction module, and the following adjacent feature extraction module of the second ASPP module is a 1 / 16 feature extraction module.

[0013] The preceding adjacent feature extraction module of the third ASPP module is a 1 / 16 feature extraction module, and the following adjacent feature extraction module of the third ASPP module is a 1 / 32 feature extraction module.

[0014] According to the present invention, a feature extraction network for semantic segmentation is provided, wherein the at least one ASPP module has a dilatation rate of [6, 12, 18].

[0015] According to the present invention, a feature extraction network for semantic segmentation is provided, wherein the feature extraction network for semantic segmentation is a dual-branch structure feature extraction network, and the dual-branch structure feature extraction network includes a semantic branch and a detail branch.

[0016] According to the present invention, a feature extraction network for semantic segmentation is provided, wherein the semantic branch includes at least two of the feature extraction modules and at least one of the ASPP modules.

[0017] Secondly, the present invention also provides a feature extraction method for semantic segmentation, applied to the feature extraction network for semantic segmentation, the method comprising:

[0018] Feature extraction is performed using at least two feature extraction modules and at least one ASPP module;

[0019] Wherein, any one of the at least one ASPP module is located between any two adjacent feature extraction modules of the at least two feature extraction modules;

[0020] The input to the ASPP module is the first feature output by the preceding adjacent feature extraction module, and the output of the ASPP module is the second feature obtained after processing the first feature. The input to the subsequent adjacent feature extraction module of the ASPP module is the feature obtained by adding the first feature and the second feature.

[0021] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the feature extraction method for semantic segmentation as provided in the second aspect.

[0022] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the feature extraction method for semantic segmentation as provided in the second aspect.

[0023] This invention provides a feature extraction network and method for semantic segmentation. By adding an ASPP module between at least two feature extraction modules, the ASPP module obtains the first feature from the preceding adjacent feature extraction module, processes the first feature to obtain the second feature, and then adds the first and second features and sends them to the following adjacent feature extraction module. By using the ASPP module to fuse large receptive field features and small receptive field features, the semantic segmentation network can simultaneously extract more contextual and detailed information, thereby improving the semantic segmentation effect. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is one of the structural schematic diagrams of the feature extraction network for semantic segmentation provided by the present invention;

[0026] Figure 2 This is the second schematic diagram of the feature extraction network for semantic segmentation provided by the present invention;

[0027] Figure 3 This is the third schematic diagram of the feature extraction network for semantic segmentation provided by the present invention;

[0028] Figure 4 This is a flowchart illustrating the feature extraction method for semantic segmentation provided by the present invention;

[0029] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] First, let's introduce the following:

[0032] Semantic segmentation deep learning networks can be broadly categorized into encoder-decoder structures and dual-branch parallel structures. Encoder-decoder structures consist of an encoder and a decoder. The encoder progressively downsamples the input to extract features, and the decoder progressively upsamples the encoder's output to obtain the semantic segmentation result. However, downsampling during the encoding stage leads to the loss of detail, and upsampling during the decoding stage also results in a lack of necessary detail, leading to inaccurate results and inaccurate edge details. Reducing downsampling during the encoder stage preserves more detail but results in insufficient extraction of contextual information. Therefore, some encoder-decoder structures use Atrous Spatial Pyramid Pooling (ASPP) after reducing encoder downsampling to expand the receptive field without reducing resolution, thereby extracting more contextual information. ASPP generates convolutional kernels with different receptive fields by setting different dilation rates, allowing for feature extraction. ASPP uses global average pooling to obtain the overall receptive field. However, in encoder-decoder structures, ASPP modules are generally only used in the last layer of the network's feature extraction section. In addition, to resolve the contradiction that "reducing downsampling in the encoder stage can retain more detailed information, but it will also lead to insufficient extraction of contextual information; while not reducing downsampling in the encoder stage will lead to loss of detailed information," a dual-branch parallel structure can be adopted. However, both branches in the dual-branch parallel structure will perform downsampling, which will also lead to the problem of loss of detailed information.

[0033] Related technologies have demonstrated that fusing features of different sizes or resolutions can improve the performance of semantic segmentation deep learning networks, such as the DeepLab series of networks. However, in encoder-decoder networks, the ASPP module is generally only used in the last layer of the feature extraction part. While fusing features with different receptive fields or sizes can improve the performance of semantic segmentation deep learning networks, related technologies do not explicitly specify the location where features with different receptive fields or resolutions should be fused. Furthermore, the "loss of detail information" problem exists in two-branch parallel networks, and related technologies do not use ASPP to address the issue of "how to simultaneously acquire more semantic and detail information."

[0034] This invention provides a feature extraction network and method for semantic segmentation, which enables the semantic segmentation network to extract more contextual and detailed information simultaneously, thereby improving the semantic segmentation effect.

[0035] The following is a detailed explanation based on several embodiments.

[0036] Figure 1 This is one of the structural diagrams of the feature extraction network for semantic segmentation provided by the present invention, such as... Figure 1 As shown, the feature extraction network includes:

[0037] At least two feature extraction modules and at least one ASPP module, wherein any one of the at least two ASPP modules is located between any two adjacent feature extraction modules of the at least two feature extraction modules;

[0038] The ASPP module is used to obtain the first feature output by the previous adjacent feature extraction module, perform feature extraction and pooling on the first feature, output the second feature, and add the first feature and the second feature and input them into the next adjacent feature extraction module.

[0039] Optionally, semantic segmentation may include classifying each pixel in the image and labeling it with its semantic information.

[0040] Optionally, the semantic information of an image may include high-level information implied in the image, such as meaning, description, category, and emotion.

[0041] Optionally, the semantic information of an image may include information about objects, scenes, people, and emotions.

[0042] For example, it can identify basic information such as the type and quantity of objects, the type and content of scenes, and the identity, gender, and age of people in images. It can also obtain the emotions or emotional states expressed in the images.

[0043] Optionally, semantic segmentation may include dividing an image into multiple regions, each of which is assigned a label representing its semantic information.

[0044] Optionally, semantic segmentation can be applied to fields such as autonomous driving, map making, and facial recognition.

[0045] Optionally, in the field of autonomous driving, semantic segmentation can be used to detect roads, crosswalks, pedestrians, and vehicles.

[0046] Alternatively, in remote sensing image processing, semantic segmentation can be used for land use and cover classification, etc.

[0047] Alternatively, in computer vision, semantic segmentation can be used for image segmentation and video pixel annotation, etc.

[0048] Optionally, the feature extraction network used for semantic segmentation may include fully convolutional neural networks (FCNs) and U-Net networks, etc.

[0049] Optionally, the feature extraction module can be a module within the feature extraction network.

[0050] Optionally, the feature extraction module can be used to extract pixels from an image.

[0051] Optionally, the feature extraction module may include at least feature extraction module 1 and feature extraction module 2.

[0052] Optionally, the ASPP module can be located between feature extraction module 1 and feature extraction module 2.

[0053] Optionally, the feature extraction module 1 can be located before the ASPP module.

[0054] Optionally, the ASPP module can be located before the feature extraction module 2.

[0055] Alternatively, the image can be fed into the feature extraction network first.

[0056] Optionally, feature extraction module 1 can extract the first feature of the image and then output it to ASPP module.

[0057] Optionally, the first feature may include semantic features of the image.

[0058] Optionally, the ASPP module can extract features by setting different dilation rates to generate convolutional kernels with different receptive fields.

[0059] Optionally, the ASPP module performs global average pooling on the features to obtain features of the overall receptive field.

[0060] Optionally, after obtaining the first feature, the ASPP module can further process the first feature to obtain the second feature.

[0061] Optionally, the second feature may include semantic features of the image.

[0062] Optionally, the second feature contains more semantic information than the first feature.

[0063] Optionally, a short-circuit connection can be used to add the first feature and the second feature together and output the result to the feature extraction module 2.

[0064] This invention provides a feature extraction network for semantic segmentation. By adding an ASPP module between at least two feature extraction modules, the ASPP module obtains the first feature from the preceding adjacent feature extraction module, processes the first feature to obtain the second feature, and then adds the first and second features and sends them to the following adjacent feature extraction module. By using the ASPP module to fuse large receptive field features and small receptive field features, the semantic segmentation network can simultaneously extract more contextual and detailed information, thereby improving the semantic segmentation effect.

[0065] Optionally, the ASPP module is used to perform feature extraction and / or pooling processing on the first feature.

[0066] Optionally, the ASPP module may perform feature extraction and / or pooling processing on the first feature by using a pooling layer to extract features from the first feature.

[0067] Optionally, the first feature can be subjected to dilated convolution to increase the receptive field of the first feature and obtain a second feature with a large receptive field.

[0068] Optionally, the receptive field can be the size of the area of ​​image pixels that can be captured.

[0069] Optionally, the receptive field of the features extracted by the ASPP module may be different from that of the features extracted by the feature extraction module.

[0070] Optionally, the receptive field of the first feature may be larger than that of the second feature.

[0071] Optionally, the area of ​​image pixels captured by a large receptive field can be larger than the area of ​​image pixels captured by a small receptive field.

[0072] Optionally, the area of ​​image pixels in the second feature can be larger than the area of ​​image pixels in the first feature.

[0073] Optionally, the more layers the network has, the larger the receptive field can be, and the more contextual information can be extracted.

[0074] This invention provides a feature extraction network for semantic segmentation. By utilizing the ASPP module to perform feature extraction and / or pooling on the first feature, features with a larger receptive field are obtained, enabling the extraction of more contextual information and improving the semantic segmentation effect.

[0075] Optionally, the at least two feature extraction modules sequentially include: a 1 / 4 feature extraction module, a 1 / 8 feature extraction module, a 1 / 16 feature extraction module, and a 1 / 32 feature extraction module.

[0076] Optionally, an ASPP module may be included between the 1 / 4 feature extraction module and the 1 / 8 feature extraction module.

[0077] Optionally, an ASPP module may be included between the 1 / 8 feature extraction module and the 1 / 16 feature extraction module.

[0078] Optionally, an ASPP module may be included between the 1 / 16 feature extraction module and the 1 / 32 feature extraction module.

[0079] Optionally, the 1 / 4 feature extraction module can first extract the features of the image (feature a), input feature a to the adjacent ASPP module, which can process feature a and output feature b, and then the ASPP module can add feature a and feature b and input them to the 1 / 8 feature extraction module.

[0080] Optionally, the 1 / 8 feature extraction module can further extract the feature obtained by adding feature a and feature b to obtain feature c. Feature c can be input to the adjacent ASPP module. The ASPP module can process feature c and output feature d. Then, the ASPP module can add feature c and feature d and input them to the 1 / 16 feature extraction module.

[0081] Optionally, the 1 / 16 feature extraction module can further extract the feature obtained by adding feature c and feature d to obtain feature e. Feature e can be input to the adjacent ASPP module, which can process feature e and output feature f. Then, the ASPP module can add feature e and feature f and input them to the 1 / 32 feature extraction module.

[0082] Optionally, the 1 / 32 feature extraction module can further extract the features obtained by adding features e and f to obtain feature g.

[0083] Optionally, the 1 / 4 feature extraction module can extract low-level global features from the image.

[0084] Optionally, compared to the 1 / 4 feature extraction module, the 1 / 8 feature extraction module can extract global features of the image at a deeper level.

[0085] Optionally, compared to the 1 / 8 feature extraction module, the 1 / 16 feature extraction module can extract global features of the image at a deeper level.

[0086] Optionally, compared to the 1 / 16 feature extraction module, the 1 / 32 feature extraction module can extract global features of the image at a deeper level.

[0087] Optionally, the at least one ASPP module includes one or more of the following: a first ASPP module, a second ASPP module, and a third ASPP module;

[0088] The preceding adjacent feature extraction module of the first ASPP module is a 1 / 4 feature extraction module, and the following adjacent feature extraction module of the first ASPP module is a 1 / 8 feature extraction module.

[0089] The preceding adjacent feature extraction module of the second ASPP module is a 1 / 8 feature extraction module, and the following adjacent feature extraction module of the second ASPP module is a 1 / 16 feature extraction module.

[0090] The preceding adjacent feature extraction module of the third ASPP module is a 1 / 16 feature extraction module, and the following adjacent feature extraction module of the third ASPP module is a 1 / 32 feature extraction module.

[0091] Figure 2 This is the second schematic diagram of the feature extraction network for semantic segmentation provided by the present invention, as shown below. Figure 2 As shown, the image can first be input into the 1 / 4 feature extraction module;

[0092] Optionally, the 1 / 4 feature extraction module can extract the features (feature A) of the image pixels, and then the 1 / 4 feature extraction module can output feature A to the first ASPP module. The first ASPP module can perform feature extraction processing and / or pooling processing on feature A to obtain feature B (second feature). Feature A and feature B can be fused and then input to the 1 / 8 feature extraction module.

[0093] Optionally, the 1 / 8 feature extraction module can further extract the features after the fusion of features A and B to obtain feature C. Then, the 1 / 8 feature extraction module can output feature C to the second ASPP module. The second ASPP module can perform feature extraction processing and / or pooling processing on feature C to obtain feature D (second feature). Feature C and feature D can be fused and input to the 1 / 16 feature extraction module.

[0094] Optionally, the 1 / 16 feature extraction module can further extract the features after the fusion of features C and D to obtain feature E. Then, the 1 / 16 feature extraction module can output feature E to the third ASPP module. The third ASPP module can perform feature extraction processing and / or pooling processing on feature E to obtain feature F (second feature). Feature E and feature F can be fused and input to the 1 / 32 feature extraction module.

[0095] Optionally, the 1 / 32 feature extraction module can further extract the features obtained by fusing features E and F to obtain feature G.

[0096] Optionally, a batch normalization (BN) layer ASPP structure can be added to the feature extraction network.

[0097] Optionally, BN layers can be used to normalize data and accelerate the training process.

[0098] This invention provides a feature extraction network for semantic segmentation. By embedding multiple ASPP modules between feature extraction modules, a large receptive field feature is obtained, enabling the semantic segmentation network to extract more contextual information and improve the semantic segmentation effect.

[0099] Optionally, the void ratio of the at least one ASPP module is [6, 12, 18].

[0100] Optionally, ASPP can extract features by setting different dilation rates to generate convolutional kernels with different receptive fields.

[0101] Optionally, the dilation rate in ASPP can be the spacing between elements in the convolution kernel during the convolution operation.

[0102] Optionally, the higher the hole rate, the larger the effective field of the convolution kernel can be, and the wider the receptive field can be, resulting in better contextual understanding of the input data.

[0103] Optionally, in ASPP, convolving the input features with convolution kernels of different dilation rates can produce receptive fields of different sizes, increasing the multi-scale nature of the features.

[0104] Optionally, the hole ratio in ASPP can be adjusted according to different data sizes and task requirements. Especially in visual tasks such as semantic segmentation, different hole ratios can be selected to solve the detection problem of objects of different sizes, depending on the task.

[0105] Optionally, the void ratio of the first ASPP module can be [6, 12, 18].

[0106] Optionally, the void ratio of the second ASPP module can be [6, 12, 18].

[0107] Optionally, the void ratio of the third ASPP module can be [6, 12, 18].

[0108] Optionally, the first ASPP module can perform convolution operations on the same feature map with dilation rates of 6, 12, and 18, and then perform pooling to obtain features with a large receptive field.

[0109] Optionally, the second ASPP module can perform convolution operations on the same feature map with dilation rates of 6, 12, and 18, and then perform pooling to obtain features with a large receptive field.

[0110] Optionally, the third ASPP module can perform convolution operations on the same feature map with dilation rates of 6, 12, and 18, and then perform pooling to obtain features with a large receptive field.

[0111] Optionally, the image can first be input into the 1 / 4 feature extraction module;

[0112] Optionally, the 1 / 4 feature extraction module can extract the features (feature A) of the image pixels, and then the 1 / 4 feature extraction module can output feature A to the first ASPP module. The first ASPP module can perform convolution operations on feature A with dilation rates of 6, 12 and 18 respectively, and then perform pooling to obtain feature B (second feature). Feature A and feature B can be fused and then input to the 1 / 8 feature extraction module.

[0113] Optionally, the 1 / 8 feature extraction module can further extract the features after fusing features A and B to obtain feature C. Then, the 1 / 8 feature extraction module can output feature C to the second ASPP module. The second ASPP module can perform convolution operations on feature C with dilation rates of 6, 12 and 18 respectively, and then perform pooling to obtain feature D (second feature). Feature C and feature D can be fused and input to the 1 / 16 feature extraction module.

[0114] Optionally, the 1 / 16 feature extraction module can further extract the features after fusing features C and D to obtain feature E. Then, the 1 / 16 feature extraction module can output feature E to the third ASPP module. The third ASPP module can perform convolution operations on feature E with dilation rates of 6, 12 and 18 respectively, and then perform pooling to obtain feature F (second feature). Feature E and feature F can be fused and input to the 1 / 32 feature extraction module.

[0115] Optionally, the 1 / 32 feature extraction module can further extract the features obtained by fusing features E and F to obtain feature G.

[0116] This invention provides a feature extraction network for semantic segmentation. By setting a larger dilatation rate, a larger receptive field is obtained, enabling the semantic segmentation network to extract more contextual information and improve the semantic segmentation effect.

[0117] Optionally, the feature extraction network for semantic segmentation is a dual-branch structure feature extraction network, which includes a semantic branch and a detail branch.

[0118] Optionally, the feature extraction network used for semantic segmentation can be a two-branch semantic segmentation network.

[0119] Alternatively, the dual-branch semantic segmentation network can be a semantic segmentation model using a dual encoder-decoder structure, where the two encoders of the model can extract features at different resolutions, and then the two decoders can fuse these features to generate the final semantic segmentation result.

[0120] Optionally, the semantic branch of the dual-branch feature extraction network can be used to extract semantic information from the image.

[0121] Optionally, the semantic branch of the dual-branch feature extraction network can extract global features in the image, such as the overall layout of the image, the category of objects, color, and texture.

[0122] Optionally, the detail branch of the dual-branch feature extraction network can be used to extract detailed information from the image.

[0123] Optionally, the detail branch of the dual-branch feature extraction network can extract local features in the image, such as details like the image's edges, texture, and shape.

[0124] Optionally, when an image is input into a dual-branch feature extraction network, after operations such as convolution and pooling, the semantic branch and the detail branch can respectively process the global and local features of the image.

[0125] Optionally, the semantic branch and the detail branch can merge the obtained feature maps and perform further calculations and processing on the merged feature maps to obtain more accurate semantic segmentation results.

[0126] Optionally, a two-branch semantic segmentation network may include the following steps when extracting features:

[0127] Step 101, encoder feature extraction;

[0128] Optionally, after inputting an image into a two-branch semantic segmentation network, it can be processed by an encoder for feature extraction and abstraction to obtain a series of abstract feature maps.

[0129] Step 102, Feature fusion;

[0130] Optionally, the feature maps of each layer obtained in the encoder can be fused with the feature maps of the corresponding layers in the decoder to obtain feature maps with representational capabilities, while retaining more positional and detailed information.

[0131] Step 103, Deconvolution of the decoder;

[0132] Optionally, during the decoding process, deconvolution operations can be performed to restore the features to the size of the original image through upsampling and padding operations, that is, a mapping from high-level abstract features to low-level abstract features, thus completing image restoration.

[0133] Step 104, Feature Adjustment;

[0134] Optionally, since the feature map in the encoder loses some positional and detailed information after abstraction, the feature map can be adjusted and detailed information added in the decoder to ensure the quality and accuracy of the final segmentation result.

[0135] This invention provides a feature extraction network for semantic segmentation. By employing a dual-branch structure feature extraction network, the semantic segmentation network can extract more contextual information, thereby improving the semantic segmentation effect.

[0136] Optionally, the semantic branch includes at least two of the feature extraction modules and at least one of the ASPP modules.

[0137] Optionally, ASPP modules can be embedded in semantic branches.

[0138] Figure 3 This is the third schematic diagram of the feature extraction network for semantic segmentation provided by the present invention, as shown below. Figure 3 As shown, the detailed branches can be sequentially divided into 1 / 2 feature extraction modules, 1 / 4 feature extraction modules, and 1 / 8 feature extraction modules;

[0139] Optionally, the semantic branch may sequentially include a 1 / 4 feature extraction module, a first ASPP module, a 1 / 8 feature extraction module, a second ASPP module, a 1 / 16 feature extraction module, a third ASPP module, and a 1 / 32 feature extraction module.

[0140] Optionally, the Cascade Feature Fusion (CFF) module can fuse features from detail branches and semantic branches.

[0141] Optionally, the CFF module can fuse the features extracted by the 1 / 2 feature extraction module in the detail branch and the features extracted by the second ASPP module in the semantic branch.

[0142] Optionally, the CFF module can fuse the features extracted by the 1 / 4 feature extraction module in the detail branch with the features extracted by the third ASPP module in the semantic branch.

[0143] Optionally, the CFF module can fuse the features extracted by the 1 / 8 feature extraction module in the detail branch and the features extracted by the 1 / 32 feature extraction module in the semantic branch.

[0144] Optionally, the image can be input to the 1 / 2 feature extraction module of the detail branch;

[0145] Optionally, the 1 / 2 feature extraction module of the detail branch can extract the features (feature M) of the image pixels, and then input the feature M into the 1 / 4 feature extraction module of the detail branch;

[0146] Optionally, the 1 / 4 feature extraction module of the detail branch can extract the features (feature N) of the image pixels, and then input the feature N into the 1 / 8 feature extraction module of the detail branch;

[0147] Optionally, the 1 / 8 feature extraction module can further extract feature N to obtain feature Q.

[0148] Optionally, the image can be input into the 1 / 4 feature extraction module of the semantic branch;

[0149] Optionally, the 1 / 4 feature extraction module of the semantic branch can extract the features (feature A) of the image pixels. Then, the 1 / 4 feature extraction module of the semantic branch can output feature A to the first ASPP module. The first ASPP module can perform convolution operations on feature A with dilation rates of 6, 12 and 18 respectively, and then perform pooling to obtain feature B (second feature). Feature A and feature B can be fused and input to the 1 / 8 feature extraction module of the semantic branch.

[0150] Optionally, the 1 / 8 feature extraction module of the semantic branch can further extract the features after the fusion of features A and B to obtain feature C. Then, the 1 / 8 feature extraction module of the semantic branch can output feature C to the second ASPP module. The second ASPP module can perform convolution operations on feature C with dilation rates of 6, 12 and 18 respectively, and then perform pooling to obtain feature D (second feature). Feature C and feature D can be fused and input to the 1 / 16 feature extraction module.

[0151] Optionally, the 1 / 16 feature extraction module can further extract the features after fusing features C and D to obtain feature E. Then, the 1 / 16 feature extraction module can output feature E to the third ASPP module. The third ASPP module can perform convolution operations on feature E with dilation rates of 6, 12 and 18 respectively, and then perform pooling to obtain feature F (second feature). Feature E and feature F can be fused and input to the 1 / 32 feature extraction module.

[0152] Optionally, the 1 / 32 feature extraction module can further extract the features obtained by fusing features E and F to obtain feature G.

[0153] Optionally, the CFF module can fuse the features M extracted by the 1 / 2 feature extraction module in the detail branch and the features D extracted by the second ASPP module in the semantic branch.

[0154] Optionally, the CFF module can fuse the features N extracted by the 1 / 4 feature extraction module in the detail branch and the features F extracted by the third ASPP module in the semantic branch.

[0155] Optionally, the CFF module can fuse the features Q extracted by the 1 / 8 feature extraction module in the detail branch and the features G extracted by the 1 / 32 feature extraction module in the semantic branch.

[0156] This invention provides a feature extraction network for semantic segmentation. By adding multiple ASPP modules to the semantic branch to obtain large receptive field features, the ASPP modules are used to fuse large and small receptive field features, and the CFF module fuses features from the detail branch and the semantic branch. This enables the semantic segmentation network to extract more contextual and detail information simultaneously, thereby improving the semantic segmentation effect.

[0157] Figure 4 This is a flowchart illustrating the feature extraction method for semantic segmentation provided by the present invention, as shown below. Figure 4 As shown, the method applied to the feature extraction network used for semantic segmentation includes the following steps:

[0158] Step 400: Feature extraction is performed using the at least two feature extraction modules and at least one ASPP module;

[0159] Wherein, any one of the at least one ASPP module is located between any two adjacent feature extraction modules of the at least two feature extraction modules;

[0160] The input to the ASPP module is the first feature output by the preceding adjacent feature extraction module, and the output of the ASPP module is the second feature obtained after processing the first feature. The input to the subsequent adjacent feature extraction module of the ASPP module is the feature obtained by adding the first feature and the second feature.

[0161] Optionally, the feature extraction module may include a 1 / 4 feature extraction module, a 1 / 8 feature extraction module, a 1 / 16 feature extraction module, and a 1 / 32 feature extraction module.

[0162] Optionally, the ASPP module may include a first ASPP module, a second ASPP module, and a third ASPP module.

[0163] Optionally, the first ASPP module can be located between the 1 / 4 feature extraction module and the 1 / 8 feature extraction module.

[0164] Optionally, the second ASPP module can be located between the 1 / 8 feature extraction module and the 1 / 16 feature extraction module.

[0165] Optionally, the third ASPP module can be located between the 1 / 16 feature extraction module and the 1 / 32 feature extraction module.

[0166] Optionally, the image can be input to the 1 / 2 feature extraction module of the detail branch;

[0167] Optionally, the 1 / 2 feature extraction module of the detail branch can extract the features (feature M) of the image pixels, and then input the feature M into the 1 / 4 feature extraction module of the detail branch;

[0168] Optionally, the 1 / 4 feature extraction module of the detail branch can extract the features (feature N) of the image pixels, and then input the feature N into the 1 / 8 feature extraction module of the detail branch;

[0169] Optionally, the 1 / 8 feature extraction module can further extract feature N to obtain feature Q.

[0170] Optionally, the image can be input into the 1 / 4 feature extraction module of the semantic branch;

[0171] Optionally, the 1 / 4 feature extraction module of the semantic branch can extract the features (feature A) of the image pixels. Then, the 1 / 4 feature extraction module of the semantic branch can output feature A to the first ASPP module. The first ASPP module can perform convolution operations on feature A with dilation rates of 6, 12 and 18 respectively, and then perform pooling to obtain feature B (second feature). Feature A and feature B can be fused and input to the 1 / 8 feature extraction module of the semantic branch.

[0172] Optionally, the 1 / 8 feature extraction module of the semantic branch can further extract the features after the fusion of features A and B to obtain feature C. Then, the 1 / 8 feature extraction module of the semantic branch can output feature C to the second ASPP module. The second ASPP module can perform convolution operations on feature C with dilation rates of 6, 12 and 18 respectively, and then perform pooling to obtain feature D (second feature). Feature C and feature D can be fused and input to the 1 / 16 feature extraction module.

[0173] Optionally, the 1 / 16 feature extraction module can further extract the features after fusing features C and D to obtain feature E. Then, the 1 / 16 feature extraction module can output feature E to the third ASPP module. The third ASPP module can perform convolution operations on feature E with dilation rates of 6, 12 and 18 respectively, and then perform pooling to obtain feature F (second feature). Feature E and feature F can be fused and input to the 1 / 32 feature extraction module.

[0174] Optionally, the 1 / 32 feature extraction module can further extract the features obtained by fusing features E and F to obtain feature G.

[0175] Optionally, the CFF module can fuse the features M extracted by the 1 / 2 feature extraction module in the detail branch and the features D extracted by the second ASPP module in the semantic branch.

[0176] Optionally, the CFF module can fuse the features N extracted by the 1 / 4 feature extraction module in the detail branch and the features F extracted by the third ASPP module in the semantic branch.

[0177] Optionally, the CFF module can fuse the features Q extracted by the 1 / 8 feature extraction module in the detail branch and the features G extracted by the 1 / 32 feature extraction module in the semantic branch.

[0178] This invention provides a feature extraction method for semantic segmentation. By adding an ASPP module between at least two feature extraction modules, the ASPP module obtains the first feature from the preceding adjacent feature extraction module, processes the first feature to obtain the second feature, and then adds the first and second features and sends them to the following adjacent feature extraction module. By using the ASPP module to fuse large receptive field features and small receptive field features, the semantic segmentation network can simultaneously extract more contextual and detailed information, thereby improving the semantic segmentation effect.

[0179] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a feature extraction method for semantic segmentation, the method including:

[0180] Feature extraction is performed using at least two feature extraction modules and at least one ASPP module;

[0181] Wherein, any one of the at least one ASPP module is located between any two adjacent feature extraction modules of the at least two feature extraction modules;

[0182] The input to the ASPP module is the first feature output by the preceding adjacent feature extraction module, and the output of the ASPP module is the second feature obtained after processing the first feature. The input to the subsequent adjacent feature extraction module of the ASPP module is the feature obtained by adding the first feature and the second feature.

[0183] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the feature extraction method for semantic segmentation provided by the above methods, the method comprising:

[0185] Feature extraction is performed using at least two feature extraction modules and at least one ASPP module;

[0186] Wherein, any one of the at least one ASPP module is located between any two adjacent feature extraction modules of the at least two feature extraction modules;

[0187] The input to the ASPP module is the first feature output by the preceding adjacent feature extraction module, and the output of the ASPP module is the second feature obtained after processing the first feature. The input to the subsequent adjacent feature extraction module of the ASPP module is the feature obtained by adding the first feature and the second feature.

[0188] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the feature extraction methods for semantic segmentation provided by the methods described above, the method comprising:

[0189] Feature extraction is performed using at least two feature extraction modules and at least one ASPP module;

[0190] Wherein, any one of the at least one ASPP module is located between any two adjacent feature extraction modules of the at least two feature extraction modules;

[0191] The input to the ASPP module is the first feature output by the preceding adjacent feature extraction module, and the output of the ASPP module is the second feature obtained after processing the first feature. The input to the subsequent adjacent feature extraction module of the ASPP module is the feature obtained by adding the first feature and the second feature.

[0192] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A feature extraction network for semantic segmentation, characterized in that, include: At least two feature extraction modules and at least one ASPP module, wherein any one of the at least two ASPP modules is located between any two adjacent feature extraction modules of the at least two feature extraction modules; The ASPP module is used to obtain the first feature output by the previous adjacent feature extraction module, process the first feature and output the second feature, and add the first feature and the second feature and input them into the next adjacent feature extraction module. The at least two feature extraction modules include, in sequence, a 1 / 4 feature extraction module, a 1 / 8 feature extraction module, a 1 / 16 feature extraction module, and a 1 / 32 feature extraction module, and the at least one ASPP module includes a first ASPP module, a second ASPP module, and a third ASPP module; The preceding adjacent feature extraction module of the first ASPP module is a 1 / 4 feature extraction module, and the following adjacent feature extraction module of the first ASPP module is a 1 / 8 feature extraction module. The preceding adjacent feature extraction module of the second ASPP module is a 1 / 8 feature extraction module, and the following adjacent feature extraction module of the second ASPP module is a 1 / 16 feature extraction module. The preceding adjacent feature extraction module of the third ASPP module is a 1 / 16 feature extraction module, and the following adjacent feature extraction module of the third ASPP module is a 1 / 32 feature extraction module.

2. The feature extraction network for semantic segmentation according to claim 1, characterized in that, The ASPP module is used to perform feature extraction and / or pooling processing on the first feature.

3. The feature extraction network for semantic segmentation according to claim 1 or 2, characterized in that, The void ratio of the at least one ASPP module is [6, 12, 18].

4. The feature extraction network for semantic segmentation according to claim 1 or 2, characterized in that, The feature extraction network used for semantic segmentation is a dual-branch structure feature extraction network, which includes a semantic branch and a detail branch.

5. The feature extraction network for semantic segmentation according to claim 4, characterized in that, The semantic branch includes at least two of the feature extraction modules and at least one of the ASPP modules.

6. A feature extraction method for semantic segmentation, characterized in that, The method, applied to the feature extraction network for semantic segmentation provided in any one of claims 1-5, comprises: Feature extraction is performed using at least two feature extraction modules and at least one ASPP module; Wherein, any one of the at least one ASPP module is located between any two adjacent feature extraction modules of the at least two feature extraction modules; The input of the ASPP module is the first feature output by the preceding adjacent feature extraction module, and the output of the ASPP module is the second feature obtained after processing the first feature. The input of the subsequent adjacent feature extraction module of the ASPP module is the feature obtained by adding the first feature and the second feature. The at least two feature extraction modules include, in sequence, a 1 / 4 feature extraction module, a 1 / 8 feature extraction module, a 1 / 16 feature extraction module, and a 1 / 32 feature extraction module, and the at least one ASPP module includes a first ASPP module, a second ASPP module, and a third ASPP module; The preceding adjacent feature extraction module of the first ASPP module is a 1 / 4 feature extraction module, and the following adjacent feature extraction module of the first ASPP module is a 1 / 8 feature extraction module. The preceding adjacent feature extraction module of the second ASPP module is a 1 / 8 feature extraction module, and the following adjacent feature extraction module of the second ASPP module is a 1 / 16 feature extraction module. The preceding adjacent feature extraction module of the third ASPP module is a 1 / 16 feature extraction module, and the following adjacent feature extraction module of the third ASPP module is a 1 / 32 feature extraction module.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the feature extraction method for semantic segmentation as described in claim 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the feature extraction method for semantic segmentation as described in claim 6.

Images