A cascaded instance segmentation method based on enhanced semantic segmentation head

Abstract

The application discloses a kind of cascade instance segmentation methods based on enhanced semantic segmentation head, comprising the following steps: extracting the multiscale feature of image, fusing multiscale feature obtains single scale feature;According to single scale feature, obtain semantic segmentation feature;According to semantic segmentation feature and multiscale feature, instance segmentation is carried out, and single instance in image is obtained, wherein, semantic segmentation feature is obtained by transformer model, convolution network model fusion.This application provides a kind of cascade instance segmentation methods based on enhanced semantic segmentation head, improves the distinguishability of semantic segmentation feature, improves the accuracy of instance.

Description

TECHNICAL FIELD

[0001] The application relates to a cascaded instance segmentation method based on an enhanced semantic segmentation head and belongs to the technical field of computer vision. BACKGROUND

[0002] Instance segmentation refers to pixel-by-pixel segmentation of single instances in an image and classification, and is widely applied to the fields of automatic driving, medical image segmentation, remote sensing image analysis and the like.

[0003] There are many existing instance segmentation methods, such as an HTC and a DSC cascaded instance segmentation method. These methods usually rely on only one full convolutional network, such as an FCN network, to extract semantic segmentation features. However, this kind of mode lacks global information, resulting in the problem that the existing methods are incomplete in instance segmentation, such as incomplete instance-intra segmentation and missing instance edge segmentation.

[0004] Therefore, it is necessary to study the instance segmentation method to solve the above problems. SUMMARY

[0005] In order to overcome the above problems, the present inventors have made an in-depth study and designed a cascaded instance segmentation method based on an enhanced semantic segmentation head, which is characterized by the following steps:

[0006] S1, extracting multi-scale features of an image, and fusing the multi-scale features to obtain single-scale features;

[0007] S2, obtaining semantic segmentation features according to the single-scale features;

[0008] S3, performing instance segmentation according to the semantic segmentation features and the multi-scale features to obtain single instances in the image.

[0009] In a preferred embodiment, in S1, the multi-scale features in the image are extracted by a feature extractor stacked with a feature pyramid.

[0010] In a preferred embodiment, in S1, the multi-scale features are fused by the following manner:

[0011] The multi-scale features extracted by the feature extractor are input into the feature pyramid, a 1x1 convolution is arranged after each scale feature of the feature pyramid, the high-layer features are up-sampled, the low-layer features are down-sampled, all the features output by the feature pyramid are fixed to a unified scale, and then the features of the unified scale are fused to obtain the single-scale features.

[0012] In a preferred embodiment, in S2, the single-scale features are input into the enhanced semantic segmentation head, and the semantic segmentation features are output by the enhanced semantic segmentation head,

[0013] The enhanced semantic segmentation head comprises a segmentation model, a transformer model, a convolution network model and a convolution layer,

[0014] The segmentation model is used to divide the input single-scale feature into multiple blocks, and each divided block is input into the transformer model,

[0015] The transformer model generates global context features x g from the input single-scale feature divided blocks.

[0016] The convolution network model generates spatial context features x s from the input single-scale feature.

[0017] After the global context features x g and the spatial context features x s are fused, the semantic segmentation features are generated through the convolution layer.

[0018] In a preferred embodiment, in S2, the convolution network model is FCN.

[0019] In a preferred embodiment, in S3, a single instance in the image is represented by a bounding box and an instance mask, and the instance segmentation is realized by a cascaded predictor, which is a multi-stage paradigm structure, and the output of the previous stage is used to train the bounding box b t and the instance mask m t of the current stage, which can be represented as:

[0020]

[0021]

[0022]

[0023]

[0024] wherein F represents multi-scale features, x en represents semantic segmentation features, t represents different stages, represents the bounding box feature of the t stage, represents the instance mask feature of the t stage, P(·) represents a pooling function, B t represents the bounding box predictor of the t stage, M t represents the instance mask predictor of the t stage, b t represents the bounding box of the t stage, m t represents the instance mask of the t stage.

[0025] In a preferred embodiment, during the training of the enhanced semantic segmentation head and the cascaded predictor, a classification supervised training process is added, which refers to a multi-label training supervised by the classes of all instances in the image.

[0026] In a preferred embodiment, in the classification supervised training, the loss function is set as:

[0027]

[0028] wherein, is a semantic segmentation loss, is a multi-label classification loss; t represents different stages of the cascaded predictor, and T is the total number of stages, is a cross-entropy loss of the bounding box of the t stage of the cascaded predictor; is a cross-entropy loss of the instance mask of the t stage of the cascaded predictor, and alpha and beta are weight coefficients, and lambda t is the training weight of different stages.

[0029] In addition, the present application also provides an electronic device comprising:

[0030] at least one processor; and

[0031] a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned method.

[0032] In addition, the present application also provides a computer readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the above-mentioned method.

[0033] The present application has the beneficial effects including:

[0034] (1) The cascaded instance segmentation method based on the enhanced semantic segmentation head provided by the present application utilizes the global modeling capability of the Transformer to improve the distinguishability of the semantic segmentation features, can be integrated with most existing cascaded instance methods, and improves the performance thereof;

[0035] (2) The cascaded instance segmentation method based on the enhanced semantic segmentation head provided by the present application is easy to configure into other cascaded instance networks, and only a small amount of parameter quantity and calculation quantity are added. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart of the cascaded instance segmentation method based on the enhanced semantic segmentation head according to a preferred embodiment of the present application is shown;

[0037] Figure 2 Fig. 1 shows a schematic diagram of a process for fusing multi-scale features in a cascaded instance segmentation method based on an enhanced semantic segmentation head according to a preferred embodiment of the present application. DETAILED DESCRIPTION

[0038] The present application will be further described by the accompanying drawings and examples. The features and advantages of the present application will become more apparent from these descriptions.

[0039] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations. Unless specifically stated otherwise, the present application is not to be limited by the details of construction, arrangement, or order of the acts in the figures.

[0040] According to the present application, a cascaded instance segmentation method based on an enhanced semantic segmentation head is provided, characterized in that, as shown in Figure 1 the method comprises the following steps:

[0041] S1, extracting multi-scale features of an image, fusing the multi-scale features to obtain single-scale features;

[0042] S2, obtaining semantic segmentation features according to the single-scale features;

[0043] S3, performing instance segmentation according to the semantic segmentation features and the multi-scale features to obtain single instances in the image.

[0044] In S1, the multi-scale features in the image are extracted by a feature extractor stacked with a feature pyramid;

[0045] In the present application, the specific structure of the feature extractor is not limited, and can be any feature extractor used for instance segmentation, for example, a general ResNet-50, ResNet-101, or a more advanced Swin-Transformer can be selected.

[0046] The feature pyramid (Feature Pyramid Network) is a basic component commonly used in detecting objects of different scales, and its structure is not described in detail in the present application.

[0047] Preferably, according to experience, the output of the feature extractor stacked with the feature pyramid is set to 5 layers of multi-scale features, and the step lengths of the 5 layers of multi-scale features relative to the original image are 2, 4, 8, 16, and 32, respectively.

[0048] Further, the fusing of the multi-scale features is realized by the following way, as shown in Figure 2

[0049] ​The multi-scale features are input into a feature pyramid, a 1x1 convolution is arranged after each scale feature of the feature pyramid, an up-sampling operation is performed on high-level features, and a down-sampling operation is performed on low-level features, so that all features output by the feature pyramid are fixed to a unified scale, then the unified scale features are fused to obtain single scale features, which can be expressed as:

[0050]

[0051] wherein F represents single scale features obtained after fusion, P i represents multi-scale features input into the feature pyramid, subscript i represents different layers of the feature pyramid, Emb i (·) represents an embedding function of the multi-scale features P i , and Samp i (·) represents a sampling function of the multi-scale features P i .

[0052] According to the present application, in S2, the single scale features are input into an enhanced semantic segmentation head, semantic segmentation features are output by the enhanced semantic segmentation head,

[0053] The enhanced semantic segmentation head comprises a segmentation model, a transformer model, a convolutional network model and a convolutional layer,

[0054] wherein the segmentation model is used to segment the input single scale features into a plurality of blocks to obtain single scale feature segmentation blocks, and each single scale feature segmentation block is input into the transformer model,

[0055] The transformer model generates global context features x g according to the input single scale feature segmentation blocks.

[0056] The convolutional network model generates spatial context features x s according to the input single scale features.

[0057] After the global context features x g and the spatial context features x s are fused, the semantic segmentation features are generated by the convolutional layer.

[0058] Preferably, the segmentation model segments the single scale features F into SxS blocks, and the segmented blocks are represented as The features are input into the Transformer model.

[0059] The transformer model is a model using an attention mechanism to improve the training speed of the model, and the specific structure can be referred to in the paper Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need [J]. arXiv, 2017. The self-attention mechanism in the transformer is calculated as follows:

[0060]

[0061] Wherein are the query, key and value of the input feature respectively, d k represents the dimension of the query and the key, and the output of the transformer model is the global context feature x g ∈R S×S×C The global context feature is generated by the transformer model, so that the neural network can pay more attention to the internal area of the instance and the main part of the instance, which is helpful for the recognition of the whole instance.

[0062] The convolutional network model can be any known convolutional model, such as R-CNN, ResNet, etc., preferably the convolutional network model is a fully convolutional network FCN, more preferably a fully convolutional network composed of four consecutive convolutions, which can better eliminate the semantic gap between different scale features of the feature pyramid, and encode the spatial context information, so that the network pays more attention to the detail part of the image, thereby generating spatial context features.

[0063] After the global context feature x g is fused with the spatial context feature x s , the semantic segmentation feature can be generated by a 1x1 convolutional layer, which can be represented as:

[0064] x en =Emb(Up(x g )+x s )

[0065] Wherein, Up(·) and Emb(·) represent the up-sampling function and the embedding function respectively.

[0066] In the present application, the semantic segmentation feature not only contains spatial context, but also contains global context, thereby guiding the current position to reference the context information for segmentation, fully utilizing the global modeling capability of the transformer, and improving the distinguishability of the semantic segmentation feature, and the enhanced semantic segmentation feature obtained by fusing the above two features not only pays attention to the details of the image, but also pays attention to the whole instance.

[0067] In S3, a single instance in the image is represented by a bounding box and an instance mask, and the instance segmentation can be implemented by any known instance segmentation predictor, such as the predictor designed in HTC, DSC.

[0068] In a preferred embodiment, the instance segmentation is implemented by a cascaded predictor, which is a multi-stage paradigm structure, and the output of the previous stage is used to train the bounding box b t and the instance mask m t , which can be represented as:

[0069]

[0070]

[0071]

[0072]

[0073] wherein F represents a multi-scale feature, x en represents a semantic segmentation feature, t represents different stages, represents the bounding box feature of stage t, represents the instance mask feature of stage t, P(·) represents a pooling function, B t represents the bounding box predictor of stage t, M t represents the instance mask predictor of stage t, b t represents the bounding box of stage t, m t represents the instance mask of stage t.

[0074] In a preferred embodiment, when training the enhanced semantic segmentation head and the cascaded predictor, a classification supervision training process is added, which is a multi-label training supervised by the classes of all instances in the image, so as to better train the Transformer model and enable the Transformer model to learn more semantic information.

[0075] More preferably, in the classification supervision training, the loss function is set as:

[0076]

[0077] wherein, is a semantic segmentation loss, is a multi-label classification loss; t represents different stages of the cascaded predictor, and T is the total number of stages, is a cross-entropy loss of the bounding box of stage t of the cascaded predictor; is a cross-entropy loss of the instance mask of stage t of the cascaded predictor, and α and β are weight coefficients, and λt are training weights for different stages.

[0078] Further, a represents semantic segmentation loss weight, and β represents multi-label classification loss weight. In a preferred embodiment, a = 0.2, and β = 3.

[0079] In a preferred embodiment, 3 stages are set in the classification supervised training, and the training weight of each stage is set as λ = [1, 0.5, 0.25].

[0080] The various embodiments of the methods described above in the present application can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip system (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0081] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0082] In the context of the present application, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0083] To provide for interaction with a user, the methods and apparatus described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0084] The methods and apparatus described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0085] The computer system can include clients and servers. This description uses the term server and client in a generic sense, and these terms are used interchangeably. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server, or VPS for short). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0086] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0087] Embodiment

[0088] Embodiment 1

[0089] The MS COCO 2017 and the MS COCO stuff dataset are used for instance segmentation experiments. The dataset is a general dataset for instance segmentation tasks, and the MS COCO refers to (ECCV|Microsoft COCO:Common Objects in Context. European Conference on Computer Vision ((github.com)), which contains 118K training images and 5K verification images, and the total number of instance categories is 80. The COCO stuff is the semantic segmentation annotation corresponding to the MS COCO.

[0090] The instance segmentation is performed by the following steps:

[0091] S1, extracting multi-scale features of the image, and fusing the multi-scale features to obtain single-scale features;

[0092] S2, obtaining semantic segmentation features according to the single-scale features;

[0093] S3, performing instance segmentation according to the semantic segmentation features and the multi-scale features to obtain single instances in the image.

[0094] In S1, the feature extractor is ResNet-50 and ResNet-101 respectively, and the output of the feature extractor stacked with the feature pyramid is set to 5 layers of multi-scale features. The step lengths of the 5 layers of multi-scale features relative to the original image are 2, 4, 8, 16 and 32 respectively.

[0095] The fusion multi-scale feature can be expressed as:

[0096]

[0097] In S2, the single-scale feature is input into the enhanced semantic segmentation head, and semantic segmentation features are output by the enhanced semantic segmentation head,

[0098] The enhanced semantic segmentation head includes a segmentation model, a transformer model, a convolution network model, and a convolution layer. The convolution network model is a full convolution network composed of four consecutive convolutions.

[0099] In S3, the cascaded predictor is a multi-stage paradigm structure, and the output of the previous stage is used to train the present stage boundary box b t and instance mask m t , expressed as:

[0100]

[0101]

[0102]

[0103]

[0104] In training the enhanced semantic segmentation head and the cascaded predictor, a classification supervision training process is added, and in the classification supervision training, the loss function is set as:

[0105]

[0106] Wherein, α=0.2, β=3, λ=[1, 0.5, 0.25].

[0107] Comparative Example

[0108] Comparative Example 1

[0109] Instance segmentation is performed with the same dataset as in Example 1, except that the HTC method is used, which is presented in the literature “Chen, Kai, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy and Dahua Lin. Hybrid Task Cascade for Instance Segmentation.” 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019): 4969-4978.”, wherein the feature extractor likewise uses ResNet-50 and ResNet-101.

[0110] Comparative Example 2

[0111] Instance segmentation is performed with the same dataset as in Example 1, except that the DSC method is used, which is presented in the literature “Ding, Hao, Siyuan Qiao, Alan Loddon Yuille and Wei Shen. Deeply Shape-guided Cascade for Instance Segmentation.” 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 8274-8284.”, wherein the feature extractor likewise uses ResNet-50 and ResNet-101.

[0112] Experimental Example

[0113] The average precision (AP) is used as a performance evaluation indicator, which calculates the average precision for all classes and all IoU thresholds. For the bounding box and instance mask, AP can be divided into bounding box AP (AP b ) and instance mask AP (AP m ); for instance mask AP, AP for different IoU thresholds is AP 50 , AP 75 , AP for different size instances is AP S , AP M , AP L .

[0114] The results of Comparative Example 1 and Comparative Examples 1 and 2 are shown in Table 1.

[0115] Table 1

[0116]

[0117] As can be seen from Table 1, the results in Example 1 are better than those of other example segmentation methods.

[0118] In the description of the present application, it should be noted that the terms "upper", "lower", "inner", "outer", "front", "back" and the like indicate the orientation or positional relationship based on the working state of the present application, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third", "fourth" are only for the purpose of description and cannot be understood as indicating or implying relative importance.

[0119] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0120] The above describes the present application in combination with the preferred embodiments, but these embodiments are only exemplary and are only for illustrative purposes. On this basis, various substitutions and improvements can be made to the present application, which all fall within the scope of protection of the present application.

Claims

1. A cascaded instance segmentation method based on an enhanced semantic segmentation head, characterized in that, The method comprises the following steps: S1, extracting multi-scale features of the image, and fusing the multi-scale features to obtain single-scale features; S2, obtaining semantic segmentation features according to the single-scale features; S3, performing instance segmentation according to the semantic segmentation features and the multi-scale features to obtain single instances in the image; In S2, the single-scale features are input into an enhanced semantic segmentation head, and the semantic segmentation features are output by the enhanced semantic segmentation head, and the convolutional network model is FCN, The enhanced semantic segmentation head comprises a segmentation model, a transformer model, a convolutional network model and a convolutional layer, The segmentation model is used to divide the input single-scale features into multiple blocks, and each divided block is input into the transformer model, the transformer model generates global context features x from the single-scale feature segmentation block input g ; The convolutional network model generates spatial context features x from the input single-scale features s ; The global context feature x g The spatial context feature x s After fusion, semantic segmentation features are generated by a convolutional layer.

2. The cascaded instance segmentation method based on the enhanced semantic segmentation head according to claim 1, wherein In S1, the multi-scale features in the image are extracted by stacking a feature pyramid on a feature extractor.

3. The cascaded instance segmentation method based on the enhanced semantic segmentation head according to claim 1, wherein In S1, the fusion of the multi-scale features is realized by the following method: The multi-scale features extracted by the feature extractor are input into a feature pyramid, a 1x1 convolution is set after each scale feature of the feature pyramid, an up-sampling operation is performed on the high-level features, and a down-sampling operation is performed on the low-level features, so that all features output by the feature pyramid are fixed to a unified scale, and then the features of the unified scale are fused to obtain single-scale features.

4. The cascaded instance segmentation method based on the enhanced semantic segmentation head according to claim 1, wherein In S3, a single instance in the image is represented by a bounding box and an instance mask, the instance segmentation is achieved by a cascaded predictor, which is a multi-stage paradigm structure, using the output of the previous stage to train the bounding box b t and the instance mask m t , can be represented as: where F denotes multi-scale features, x en denotes semantic segmentation features, t denotes different stages, denotes the bounding box features of stage t, denotes the instance mask features of stage t, P(·) denotes a pooling function, B t denotes the bounding box predictor of stage t, M t denotes the instance mask predictor of stage t, b t denotes the bounding box of stage t, m t denotes the instance mask of stage t.

5. The cascaded instance segmentation method based on the enhanced semantic segmentation head according to claim 4, wherein During the training of the enhanced semantic segmentation head and the cascaded predictor, a classification supervision training process is added, and the classification supervision training refers to a multi-label training with the classes of all instances in the image as the supervision objects.

6. The cascaded instance segmentation method based on the enhanced semantic segmentation head according to claim 5, wherein In the classification supervision training, the loss function is set as: wherein, is a semantic segmentation loss, is a multi-label classification loss; t denotes different stages of the cascaded predictor, T is the total number of stages, is a cross-entropy loss for bounding boxes of the cascaded predictor stage t; is a cross-entropy loss for instance masks of the cascaded predictor stage t, a and b are weight coefficients, l t is the training weight of different stages.

7. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

8. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-6.

Images