An image recognition segmentation method

Abstract

The application relates to an image recognition segmentation method. An input static picture is subjected to a layered Transformer encoder structure, a PfAAM, an SPPCSPC module and a feature enhancement module are added, feature extraction and feature enhancement are carried out on the picture, and a plurality of features with different resolutions are obtained. An All-MLP decoder structure is used to fuse the multi-level features to generate a final semantic segmentation mask. The application comprehensively considers global information and local information to obtain more accurate prediction results. Experiments prove that the application has the advantages of high precision and low cost.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to an image recognition and segmentation method. Background Technology

[0002] Image segmentation has always been a fundamental problem in the field of computer vision. Image segmentation can be divided into semantic segmentation based on semantic labels or instance segmentation of single objects. Generally speaking, the pixel segmentation of a target actually places more stringent requirements on classification. Many image segmentation algorithms have been used for a long time, including earlier algorithms such as thresholding, histogram bundling, k-clustering, and watershed methods; and more advanced algorithms such as active contouring, graph reduction, conditional and Markov random fields, and sparsity-based methods.

[0003] In recent years, deep learning models have yielded novel methods for generating image segmentation models with significantly improved performance, achieving top accuracy on popular benchmarks. Convolutional Neural Networks (CNNs) are among the most successful and widely used architectures in the deep learning community, particularly in computer vision tasks. A key computational advantage of CNNs is that all receptive domains within a single layer share weights, resulting in a significantly fewer number of parameters compared to fully connected neural networks. However, existing image recognition and segmentation methods suffer from low accuracy, computational complexity, and difficulty in segmenting small-scale data. Summary of the Invention

[0004] To overcome at least one deficiency in the prior art, this application provides an image recognition and segmentation method.

[0005] Firstly, an image recognition and segmentation method is provided, including:

[0006] Obtain the model training dataset; the samples in the model training dataset are images containing the target.

[0007] An image recognition and segmentation model is constructed. The image recognition and segmentation model includes a hierarchical Transformer encoder and an MLP decoder. The hierarchical Transformer encoder includes four Transformer Blocks, a PfAAM attention module, an SPPCSPC module, and a feature enhancement module. The hierarchical Transformer encoder is used to extract and enhance features from the image, resulting in multiple features with different resolutions. The MLP decoder is used to fuse multiple features with different resolutions and finally outputs the class probability distribution of each pixel in the image.

[0008] The image recognition and segmentation model is trained based on the model training dataset to obtain the trained image recognition and segmentation model.

[0009] The image to be recognized is input into the trained image recognition and segmentation model to obtain the image recognition and segmentation result.

[0010] In one embodiment, the four Transformer Blocks are Transformer Block1, Transformer Block2, Transformer Block3, and Transformer Block4, and the PfAAM attention module is set between Transformer Block1 and Transformer Block2;

[0011] The output of Transformer Block 4 is input to the SPPCSPC module. The output of the SPPCSPC module and the output of Transformer Block 3 are both input to the feature enhancement module for feature enhancement operation, and the outputs are two enhanced features.

[0012] The outputs of Transformer Block 1, Transformer Block 2, and the two enhanced features are all input into the MLP decoder.

[0013] In one embodiment, the feature enhancement module is used to implement the following functions:

[0014] The output of the SPPC module is denoted as feature B4, and the output of Transformer Block 3 is denoted as feature B3;

[0015] Upsample feature B4 to obtain feature B4';

[0016] Adding feature B4' to feature B3 yields feature A;

[0017] Adding feature A to feature B4' yields feature A1;

[0018] We downsample feature A to obtain feature A';

[0019] Downsample feature A1 to obtain feature A1';

[0020] Adding feature A', feature A1', and feature B4 together yields feature B;

[0021] Feature A1 and feature B are the enhanced versions of the two features.

[0022] In one embodiment, the loss functions used during model training include the cross-entropy loss function and the Dice loss function.

[0023] Secondly, an image recognition and segmentation apparatus is provided, comprising:

[0024] The dataset acquisition module is used to acquire the model training dataset; the samples in the model training dataset are images containing the target.

[0025] The model building module is used to construct an image recognition and segmentation model. The image recognition and segmentation model includes a hierarchical Transformer encoder and an MLP decoder. The hierarchical Transformer encoder includes four Transformer Blocks, a PfAAM attention module, an SPPCSPC module, and a feature enhancement module. The hierarchical Transformer encoder is used to extract and enhance features from the image, obtaining multiple features with different resolutions. The MLP decoder is used to fuse multiple features with different resolutions and finally output the class probability distribution of each pixel in the image.

[0026] The model training module is used to train the image recognition and segmentation model based on the model training dataset to obtain the trained image recognition and segmentation model.

[0027] The recognition and segmentation module is used to input the image to be recognized into the trained image recognition and segmentation model to obtain the image recognition and segmentation result.

[0028] In one embodiment, the four Transformer Blocks are Transformer Block1, Transformer Block2, Transformer Block3, and Transformer Block4, and the PfAAM attention module is set between Transformer Block1 and Transformer Block2;

[0029] The output of Transformer Block 4 is input to the SPPCSPC module. The output of the SPPCSPC module and the output of Transformer Block 3 are both input to the feature enhancement module for feature enhancement operation, and the outputs are two enhanced features.

[0030] The outputs of Transformer Block 1, Transformer Block 2, and the two enhanced features are all input into the MLP decoder.

[0031] In one embodiment, the feature enhancement module is used to implement the following functions:

[0032] The output of the SPPC module is denoted as feature B4, and the output of Transformer Block 3 is denoted as feature B3;

[0033] Upsample feature B4 to obtain feature B4';

[0034] Adding feature B4' to feature B3 yields feature A;

[0035] Adding feature A to feature B4' yields feature A1;

[0036] We downsample feature A to obtain feature A';

[0037] Downsample feature A1 to obtain feature A1';

[0038] Adding feature A', feature A1', and feature B4 together yields feature B;

[0039] Feature A1 and feature B are the enhanced versions of the two features.

[0040] In one embodiment, the loss functions used during model training include the cross-entropy loss function and the Dice loss function.

[0041] Thirdly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the image recognition and segmentation method described above.

[0042] Fourthly, a computer program product is provided, including a computer program / instructions, which, when executed by a processor, implement the image recognition and segmentation method described above.

[0043] Compared with existing technologies, this application has the following advantages: This application uses a hierarchical Transformer encoder structure to process the input static image, and through the addition of PfAAM (Parameter-free Average Attention Module), SPPSPC (SPP Cross Stage Partial Channel), and a feature enhancement module (PTWNet), it extracts and enhances features of the image, obtaining multiple features with different resolutions. Then, an All-MLP decoder structure is used to fuse these multi-level features to generate the final semantic segmentation mask. This application comprehensively considers both global and local information, resulting in more accurate prediction results. Experimental verification shows that this application has advantages such as high accuracy and low cost. Attached Figure Description

[0044] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings:

[0045] Figure 1 A flowchart of an image recognition and segmentation method according to an embodiment of this application is shown;

[0046] Figure 2 A structural block diagram of an image recognition and segmentation model according to an embodiment of this application is shown;

[0047] Figure 3 A structural block diagram of the SPPCSPC module according to an embodiment of this application is shown;

[0048] Figure 4 A structural block diagram of a feature enhancement module according to an embodiment of this application is shown;

[0049] Figure 5 A structural block diagram of an image recognition and segmentation apparatus according to an embodiment of this application is shown. Detailed Implementation

[0050] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0051] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution according to this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0052] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0053] This application provides an image recognition and segmentation method. Figure 1 A flowchart of an image recognition and segmentation method according to an embodiment of this application is shown. See also: Figure 1 The methods include:

[0054] Step S1: Obtain the model training dataset; the samples in the model training dataset are images containing the target.

[0055] Here, the model training dataset can be the publicly available ADE20K dataset or the Cityspace dataset.

[0056] The ADE20K dataset is an annotated image dataset of indoor, outdoor, and natural scenes. It contains 20,210 training images and 2,000 validation images. The dataset has 150 categories: 35 east-west categories (walls, sky, roads) and 115 discrete object categories (cars, people, tables). Annotated pixels across these 150 categories comprise 92.75% of all pixels in the dataset, with the east-west category (amorphous background regions) accounting for 60.92% and the discrete object category (discrete objects) accounting for 31.83%.

[0057] The Cityspace dataset is an image dataset containing 5,000 images of driving scenes in urban environments. It includes 2,975 training images and 500 validation images. The dataset contains high-resolution images from 50 German cities, covering street scenes under different seasons, weather conditions, and traffic situations. The images are labeled into 30 categories, including roads, buildings, pedestrians, and vehicles, and can be used to train and evaluate the performance of semantic segmentation algorithms.

[0058] Step S2: Construct an image recognition and segmentation model; Figure 2 A structural block diagram of an image recognition and segmentation model according to an embodiment of this application is shown. See also: Figure 2 The image recognition and segmentation model includes a hierarchical Transformer encoder and an MLP decoder. The hierarchical Transformer encoder includes four Transformer Blocks, a PfAAM attention module, an SPPCSPC module, and a feature enhancement module. The hierarchical Transformer encoder is used to extract and enhance features from the image, resulting in multiple features with different resolutions. The MLP decoder is used to fuse multiple features with different resolutions and finally outputs the class probability distribution of each pixel in the image.

[0059] Here, a hierarchical Transformer encoder is used to generate high-resolution coarse features and low-resolution fine features, and the MLP decoder can use a lightweight All-MLP to fuse these multi-level features to generate the final prediction result.

[0060] Step S3: Train the image recognition and segmentation model based on the model training dataset to obtain the trained image recognition and segmentation model.

[0061] Step S4: Input the image to be recognized into the trained image recognition and segmentation model to obtain the image recognition and segmentation result.

[0062] In one embodiment, see Figure 2The four Transformer Blocks are Transformer Block1, Transformer Block2, Transformer Block3, and Transformer Block4, and the PfAAM attention module is set between Transformer Block1 and Transformer Block2.

[0063] The output of Transformer Block 4 is input to the SPPCSPC module. The output of the SPPCSPC module and the output of Transformer Block 3 are both input to the feature enhancement module for feature enhancement operation, and the outputs are two enhanced features.

[0064] The outputs of Transformer Block 1, Transformer Block 2, and the two enhanced features are all input into the MLP decoder.

[0065] In this embodiment, before the image is input to the hierarchical Transformer encoder, it is divided into 4×4 patches using the Patches to Tokens method. These patches serve as input to the hierarchical Transformer encoder, and the original image resolution is obtained through four TransformerBlocks. Multi-level features at the location.

[0066] The image obtains its original resolution after passing through Transformer Block 1. The feature map after passing through the PfAAM attention module yields a more detailed feature representation in the direction.

[0067] Transformer Block 4 outputs the original image resolution. The feature map after that is then processed by the SPPC module. Figure 3The diagram illustrates the structural block diagram of the SPPCSPC module according to an embodiment of this application. The Spatial Pyramid Pooling (SPP) in the SPPCSPC module performs pooling operations on the feature map at sizes of 5×5, 9×9, and 13×13. For the 5×5 scale, the spatial size of the pooling sub-image is (H-1)×(W-1), where H and W are the height and width of the original image, respectively, and each pixel is surrounded by a 2-pixel border. For the 9×9 scale, the spatial size of the pooling sub-image is (H-3)×(W-3), and each pixel is surrounded by a 4-pixel border. For the 13×13 scale, the spatial size of the pooling sub-image is (H-6)×(W-6), and each pixel is surrounded by a 6-pixel border. By generating multi-scale pooling sub-images, feature information at different scales is obtained. These feature sub-images at different scales are then stacked together to form a multi-channel feature representation. Finally, the concatenated multi-scale feature representations are passed through a convolutional layer to obtain the final SPP feature representation Y1. Then, the feature map obtained by convolving Y1 and the original input feature map are convolved together to obtain a new feature Y2, thus obtaining the final feature representation.

[0068] In one embodiment, Figure 4 A structural block diagram of a feature enhancement module according to an embodiment of this application is shown. See also: Figure 4 The feature enhancement module is used to implement the following functions:

[0069] The output of the SPPC module is denoted as feature B4, and the output of Transformer Block 3 is denoted as feature B3;

[0070] Upsample feature B4 to obtain feature B4';

[0071] Adding feature B4' to feature B3 yields feature A;

[0072] Adding feature A to feature B4' yields feature A1;

[0073] We downsample feature A to obtain feature A';

[0074] Downsample feature A1 to obtain feature A1';

[0075] Adding feature A', feature A1', and feature B4 together yields feature B;

[0076] Feature A1 and feature B are the enhanced versions of the two features.

[0077] In this embodiment, the feature enhancement module can fuse features from different scales to enhance the expressive power of the features and obtain enhanced feature representations.

[0078] In one embodiment, see Figure 2The MLP decoder can use a lightweight All-MLP, including a first MLP (Multilayer Perceptron) and a second MLP. Here, multiple features with different resolutions output by the hierarchical Transformer encoder are input into the first MLP for channel dimension unification, upsampling, and concatenation to obtain fused features. The fused features are then subjected to cross-channel convolution to obtain feature maps. The feature maps are then input into the second MLP, which is a classification head used to map the feature maps to specific categories. Finally, a softmax function is used to generate the category probability distribution for each pixel.

[0079] In one embodiment, the loss functions used during model training include the cross-entropy loss function and the Dice loss function.

[0080] Wherein, the cross-entropy loss function L:

[0081]

[0082] Where y is the actual label value. This represents the predicted probability value.

[0083] Dice loss function:

[0084]

[0085] The total loss function is Ltotal:

[0086] L_total = L + DiceLoss

[0087] Here, the parameters of the image recognition and segmentation network are adjusted by the value of the total loss function. When the value of the target loss function reaches a minimum, the update of the image recognition and segmentation network parameters is stopped, and finally the trained image recognition and segmentation model is obtained.

[0088] Based on the same inventive concept as the image recognition and segmentation method, this embodiment also provides a corresponding image recognition and segmentation device. Figure 5 A structural block diagram of an image recognition and segmentation apparatus according to an embodiment of this application is shown. See also: Figure 5 The device includes:

[0089] The dataset acquisition module 51 is used to acquire the model training dataset; the samples in the model training dataset are images containing targets.

[0090] Model building module 52 is used to build an image recognition and segmentation model. The image recognition and segmentation model includes a hierarchical Transformer encoder and an MLP decoder. The hierarchical Transformer encoder includes four Transformer Blocks, a PfAAM attention module, an SPPCSPC module, and a feature enhancement module. The hierarchical Transformer encoder is used to extract and enhance features from the image to obtain multiple features with different resolutions. The MLP decoder is used to fuse multiple features with different resolutions and finally output the class probability distribution of each pixel in the image.

[0091] The model training module 53 is used to train the image recognition and segmentation model based on the model training dataset to obtain the trained image recognition and segmentation model.

[0092] The recognition and segmentation module 54 is used to input the image to be recognized into the trained image recognition and segmentation model to obtain the image recognition and segmentation result.

[0093] The image recognition and segmentation device of this embodiment has the same inventive concept as the image recognition and segmentation method described above. Therefore, the specific implementation of the device can be found in the embodiment section of the image recognition and segmentation method described above, and its technical effects correspond to the technical effects of the above method. It will not be repeated here.

[0094] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the image recognition and segmentation method described above.

[0095] This application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the image recognition and segmentation method described above.

[0096] In summary, this application has the following technical effects:

[0097] This application uses a hierarchical Transformer encoder structure to process the input static image, followed by an added PfAAM attention module, SPCSPC module, and feature enhancement module to extract and enhance features, resulting in multiple features with different resolutions. An All-MLP decoder structure is then used to fuse these multi-level features to generate the final semantic segmentation mask. This application comprehensively considers both global and local information to obtain more accurate prediction results. Experimental verification shows that this application has advantages such as high accuracy and low cost.

[0098] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image recognition and segmentation method, characterized in that, include: Obtain the model training dataset; the samples in the model training dataset are images containing the target. An image recognition and segmentation model is constructed. This model includes a hierarchical Transformer encoder and an MLP decoder. The hierarchical Transformer encoder comprises four Transformer Blocks, a PfAAM attention module, an SPPCSPC module, and a feature enhancement module. The hierarchical Transformer encoder is used to extract and enhance features from the image, obtaining multiple features with different resolutions. The MLP decoder is used to fuse these multiple features with different resolutions and ultimately output the class probability distribution for each pixel in the image. The image recognition and segmentation model is trained based on the model training dataset to obtain the trained image recognition and segmentation model; The image to be recognized is input into the trained image recognition and segmentation model to obtain the image recognition and segmentation result; The four Transformer Blocks are Transformer Block1, Transformer Block2, Transformer Block3, and Transformer Block4, and the PfAAM attention module is located between Transformer Block1 and Transformer Block2. The output of Transformer Block 4 is input to the SPPCSPC module. The output of the SPPCSPC module and the output of Transformer Block 3 are both input to the feature enhancement module for feature enhancement operation, and the two enhanced features are output. The outputs of Transformer Block 1 and Transformer Block 2, as well as the two enhanced features, are all input into the MLP decoder. The feature enhancement module is used to achieve the following functions: The output of the SPCSPC module is denoted as feature B4, and the output of the Transformer Block 3 is denoted as feature B3; Upsample feature B4 to obtain feature B4'; Adding feature B4' to feature B3 yields feature A; Adding feature A to feature B4' yields feature A1; We downsample feature A to obtain feature A'; Downsample feature A1 to obtain feature A1'; Adding feature A', feature A1', and feature B4 together yields feature B; Feature A1 and feature B are the enhanced versions of the two features.

2. The method as described in claim 1, characterized in that, During model training, the loss functions used include cross-entropy loss and Dice loss.

3. An image recognition and segmentation device, characterized in that, include: The dataset acquisition module is used to acquire the model training dataset; The samples in the model training dataset are images containing targets; The model building module is used to construct an image recognition and segmentation model. The image recognition and segmentation model includes a hierarchical Transformer encoder and an MLP decoder. The hierarchical Transformer encoder includes four TransformerBlocks, a PfAAM attention module, an SPPCSPC module, and a feature enhancement module. The hierarchical Transformer encoder is used to extract and enhance features from the image, obtaining multiple features with different resolutions. The MLP decoder is used to fuse these multiple features with different resolutions and finally output the class probability distribution of each pixel in the image. The model training module is used to train the image recognition and segmentation model based on the model training dataset to obtain the trained image recognition and segmentation model. The recognition and segmentation module is used to input the image to be recognized into the trained image recognition and segmentation model to obtain the image recognition and segmentation result; The four Transformer Blocks are Transformer Block1, Transformer Block2, Transformer Block3, and Transformer Block4, and the PfAAM attention module is located between Transformer Block1 and Transformer Block2. The output of Transformer Block 4 is input to the SPPCSPC module. The output of the SPPCSPC module and the output of Transformer Block 3 are both input to the feature enhancement module for feature enhancement operation, and the two enhanced features are output. The outputs of Transformer Block 1 and Transformer Block 2, as well as the two enhanced features, are all input into the MLP decoder. The feature enhancement module is used to achieve the following functions: The output of the SPCSPC module is denoted as feature B4, and the output of the Transformer Block 3 is denoted as feature B3; Upsample feature B4 to obtain feature B4'; Adding feature B4' to feature B3 yields feature A; Adding feature A to feature B4' yields feature A1; We downsample feature A to obtain feature A'; Downsample feature A1 to obtain feature A1'; Adding feature A', feature A1', and feature B4 together yields feature B; Feature A1 and feature B are the enhanced versions of the two features.

4. The apparatus as described in claim 3, characterized in that, During model training, the loss functions used include cross-entropy loss and Dice loss.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the image recognition and segmentation method according to any one of claims 1-2.

6. A computer program product, characterized in that, Includes a computer program / instruction, which, when executed by a processor, implements the image recognition and segmentation method according to any one of claims 1-2.

Images