Example-semantic fusion segmentation method for part-level image segmentation

Abstract

The invention discloses an instance-semantic fusion segmentation method for part-level image segmentation, and the method comprises the steps: carrying out the preprocessing of an input image, carrying out the foreground target detection through a Faster R-CNN network, and obtaining candidate region features generated by a region proposal network; and then, constructing a target part segmentation branch to obtain a part segmentation mask, constraining the part segmentation mask by adopting part perception segmentation loss, and improving the segmentation precision of different parts of the target by combining the part perception segmentation loss with a plurality of pixel-level loss functions. And finally, fully training the target detection branch by using a progressive training strategy, and then training the target detection branch and the target part segmentation branch at the same time, thereby improving the network detection and segmentation quality. According to the invention, the problem that the existing image segmentation method is insufficient in target part recognition is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an instance-semantic fusion segmentation method for part-level image segmentation. Background Art

[0002] With the rapid development of computer vision and deep learning technologies, image segmentation, as one of the key tasks, has received widespread attention. Image segmentation aims to divide an image into multiple semantically meaningful regions, which is crucial for many practical applications such as medical image analysis, autonomous driving, and industrial production.

[0003] As application requirements increase, it becomes necessary not only to identify the entire target in an image but also to accurately segment its individual parts, a task known as part-level image segmentation. Part-level image segmentation combines the advantages of object detection and semantic part segmentation, enabling more detailed scene analysis and application processing.

[0004] Object detection aims to automatically identify foreground objects in images or videos, and simultaneously provide the location information (usually bounding box coordinates) and category labels of these objects. It is the basis of many other computer vision tasks, such as target tracking, and plays a key role in real-world applications. Object detection methods based on deep learning have made significant progress in recent years. According to the processing flow, they can be divided into two-stage object detection algorithms and single-stage object detection algorithms. The two-stage algorithm first generates candidate object regions through a region proposal network, and then classifies and refines the positioning of these candidate regions, such as the R-CNN series. The single-stage algorithm completes the object detection task in a single forward propagation, and directly predicts the category and location of the target through a dense grid or anchor box, such as the YOLO series and the DETR series.

[0005] Semantic part segmentation aims to perform fine-grained segmentation of objects in an image, ensuring that each part (such as the head and limbs of a person, or the wheels and windows of a car) can be accurately segmented and assigned a corresponding category label. Unlike semantic segmentation, semantic part segmentation segments individual parts within an object, rather than the entire scene. Semantic segmentation algorithms based on convolutional neural networks (CNNs) build deep learning models to automatically learn feature representations in images and use these features for pixel-level classification. This approach has made significant progress thanks to the efficiency and robustness of CNNs. Representative methods include FCN, DeepLab, and UNet. However, the local nature of convolutional filters limits access to global image information, which in turn affects segmentation performance. To overcome this limitation, recent semantic segmentation models have introduced the Transformer architecture, leveraging its powerful global context modeling capabilities to improve semantic segmentation. Representative methods include Segmenter, SegFormer, and MaskFormer.

[0006] Object detection primarily focuses on locating and identifying objects in images. It typically outputs bounding boxes and category labels, but fails to provide information about the object's internal structure. Semantic part segmentation, while capable of classifying pixels in different parts of an image, cannot distinguish between different instances of the same object category. Summary of the Invention

[0007] The purpose of the present invention is to provide an instance-semantic fusion segmentation method for part-level image segmentation, so as to solve the problem that existing image segmentation methods are insufficient in identifying target parts.

[0008] To achieve the above objectives, the present invention adopts a technical solution: an instance-semantic fusion segmentation method for part-level image segmentation, comprising:

[0009] Step 1: Build an instance-semantic fusion segmentation network based on FCOS. FCOS is used to extract multi-scale convolutional features of the image and detect foreground objects.

[0010] Step 2: Construct a part feature extraction module and use it to process the obtained multi-scale convolution features to extract the part semantic information in the image, that is, all possible target part feature maps;

[0011] Step 3: Calculate the loss between the obtained target part feature map and the real part label to provide auxiliary supervision for the extracted part semantic information;

[0012] Step 4: Build an instance mask confidence module and connect it in parallel to the object detection head. The convolution blocks in series generate the same number of instance mask confidences as the number of predicted objects.

[0013] In step 5, the target part feature map obtained in step 2 and the instance mask confidence obtained in step 4 are fused to obtain the part segmentation mask of all predicted targets.

[0014] The technical solution of the present invention also has the following characteristics:

[0015] In step 2, the obtained multi-scale convolution features are processed using the convolution, upsampling and element-wise addition operations.

[0016] Step 1 is implemented as follows:

[0017] Step 1.1: Preprocess the input image, including resizing and normalization. The process is expressed as follows:

[0018]

[0019] Where I represents the input image, resize is the size scaling operation, μ is the mean of the image, σ is the standard deviation of the image, and I′ is the image after preprocessing.

[0020] Step 1.2: The preprocessed image is passed through the backbone network for convolutional feature extraction. The backbone network is mainly composed of multiple residual blocks, which are used to extract semantic features at different levels. The process is expressed as follows:

[0021] {C1,C2,C3,C4,C5}=ResNet(I′) (2)

[0022] Among them, C1, C2, C3, C4, and C5 are the feature maps of each layer output by the backbone network. represents the feature map of layer i, d i is the number of channels;

[0023] Step 1.3: Pass C3, C4, and C5 through the feature pyramid network. The feature pyramid network introduces a top-down and horizontal connection structure to fuse feature maps at different levels, thereby obtaining multi-scale, semantically rich features. The process is expressed as follows:

[0024] P l =Conv3×3(Upsample(P l+1 )+Conv1×1(C l )),l=5,4,3 (3)

[0025] P m =Conv3×3(P m-1 ),m=6,7 (4)

[0026] Among them, Conv1×1 represents a convolution with a kernel size of 1×1, which is used to adjust the channel; Conv3×3 represents a convolution with a kernel size of 3×3, which is used to smooth the fused features; Upsample(·) is an upsampling operation, usually the nearest neighbor or bilinear interpolation. Finally, the multi-scale feature maps P3, P4, P5, P6 and P7 are obtained;

[0027] Step 1.4: Send P3, P4, P5, P6, and P7 to the target detection head to detect the foreground target in the image;

[0028] Classification prediction is used to determine the category to which the target belongs. The process is expressed as:

[0029]

[0030] Among them, σ represents the sigmoid function, Conv cls It is a number of convolutional layers used in the classification branch;

[0031] Bounding box regression is used to determine the location and size of the target bounding box. The process is expressed as:

[0032] t x,y =ReLU(Conv reg (P l (x,y)))=(l,t,r,b),l=3,4,5,6,7 (6)

[0033] Among them, ReLU represents RELU activation function, Conv reg are the convolutional layers used by the regression branch, l, t, r, b represent the distances from the center point to the left, top, right, and bottom of the bounding box;

[0034] The centrality prediction represents the distance between the target and the center point. The process is expressed as:

[0035] c x,y =σ(Conv ctr (P l (x,y))),l=3,4,5,6,7 (7)

[0036] Among them, σ represents the sigmoid function, Conv ctr are several convolutional layers used in the center branch.

[0037] In step 1.4, the target detection head contains a classification branch, a regression branch, and a centerness branch. The classification branch generates a predicted category score, the regression branch generates the target bounding box regression parameters, and the centerness branch outputs a centerness value.

[0038] Step 2 is implemented as follows:

[0039] In step 2.1, P3, P4, and P5 are fused through convolution, upsampling, and element-wise addition to generate a high-resolution, semantically rich unified feature map. The process is expressed as follows:

[0040] F fusion =CBR(P3)+Up 2 (CBR(P4))+Up 4 (CBR(P5)) (8)

[0041] Among them, CBR represents Conv, BN and ReLU operations, Up 2 and Up 4 Respectively represent upsampling by 2 times and upsampling by 4 times, F fusion It is the feature map after the fusion of P3, P4 and P5;

[0042] Step 2.2, fuse the feature map F fusion After the concatenated convolution blocks and upsampling operations, the target part feature map representing the semantic information of the image part is obtained. The process is expressed as follows:

[0043] F part =Conv(Up 2 CBR 4 )(F fusion ) (9)

[0044] Among them, CBR 4 Represents 4 consecutive Conv, BN and ReLU operations, and the size of the resulting part feature map T is (N, H / 4, W / 4), where N is the number of possible target parts, and H and W are the width and height of the original image.

[0045] Step 3 is implemented as follows:

[0046] Step 3.1, preprocess the true label of the image;

[0047] In step 3.2, an improved cross entropy loss function with weights and boundary guidance is used to make it more suitable for auxiliary supervision of part semantic information. The formula is expressed as:

[0048]

[0049] Among them, α c is the weight of different part categories, which is set to the inverse of the frequency of the category in the training set, that is:

[0050]

[0051] w iIt is the position weight used for boundary enhancement, which can be used to control the importance of the boundary area by the boundary mask generated by the edge detector:

[0052]

[0053] In step 4: construct an instance mask confidence module (IMCM), which is connected in parallel to the object detection head. Through the series of convolution blocks, an instance mask confidence feature map with the same number of predicted objects is generated. The feature map size is (M, H / 8, W / 8, N), where M is the number of predicted objects, H and W are the width and height of the original image, and N is the number of possible object parts. The process is expressed as follows:

[0054] F conf =σ(Conv(CGR 4 (P l (x,y)))),l=3,4,5,6,7 (13)

[0055] Among them, CGR represents Conv, GN and ReLU operations, and σ represents the sigmoid activation function.

[0056] Step 5 is implemented as follows:

[0057] Step 5.1, before fusion, first conf Perform upsampling interpolation operation, and F part The feature map size is aligned. The process is expressed as:

[0058] F conf '=inter 2 (F conf ) (14)

[0059] in, Represents a 2x bilinear interpolation operation, and the resulting F conf 'Size is (M,H / 4,W / 4,N), and F part consistent;

[0060] Step 5.2, and F part The multiplication and summation operation in the channel direction can be:

[0061]

[0062] F conf 'Provide the top instance information of the image, F part Providing the semantic information of the underlying parts, the two are fused to obtain the final target part segmentation result.

[0063] The beneficial effects of the present invention are:

[0064] (1) The present invention proposes a part-level image segmentation network, which is an end-to-end structure that realizes parallel processing of target detection and part segmentation. Compared with traditional target recognition methods, this method can effectively detect the part information inside the target. Compared with simple semantic segmentation, it can distinguish different instances of objects of the same category. Therefore, the present invention can be widely used in medical image analysis, autonomous driving, intelligent security monitoring and other fields, providing an effective method for the accurate segmentation of different parts in an image.

[0065] (2) The present invention uses an instance-semantic fusion strategy to predict the target part segmentation mask. Specifically, the part feature extraction module (PFEM) is used to extract the bottom-level semantic information of the part in the image, and the instance mask confidence module (IMCM) is used to obtain the top-level instance information. Finally, the instance and semantic features are fused to obtain the segmentation mask of the target part. The complementary characteristics of instance information and semantic part information are utilized to enhance the feature expression capability of the model.

[0066] (3) The present invention makes the segmentation method adaptable to a variety of application scenarios by designing an efficient network architecture and optimization strategy. Through the end-to-end network structure, instance-semantic feature fusion, and auxiliary supervision strategy for part information, the network can effectively extract multi-level features and adaptively adjust the weight distribution between different features to more accurately identify and segment different parts of the target. This fusion mechanism not only improves the segmentation accuracy, but also enhances the model's adaptability to changes in target morphology and scale, so that stable segmentation results can be obtained even in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the overall design of the instance-semantic fusion segmentation method for part-level image segmentation of the present invention.

[0068] Figure 2 It is a schematic diagram of the framework of the instance-semantic fusion segmentation method for part-level image segmentation of the present invention.

[0069] Figure 3 This is the segmentation result of the instance-semantic fusion segmentation method for part-level image segmentation of the present invention on the MVPOP dataset. DETAILED DESCRIPTION

[0070] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] Example 1

[0072] This paper aims to provide an instance-semantic fusion segmentation method for part-level image segmentation, which accurately detects and segments objects in images. This method employs an end-to-end network architecture that enables parallel processing of object detection and part segmentation. For part segmentation, a Part Feature Extraction Module (PFEM) extracts underlying semantic information about parts in the image, while an Instance Mask Confidence Module (IMCM) obtains top-level instance information. By fusing instance and semantic features, a segmentation mask for the target part is generated.

[0073] Specifically, the technical solution adopted by the present invention is as follows: first, the input image is preprocessed, and the multi-scale convolutional features of the image are extracted using FCOS, and the foreground targets are detected. Then, a part feature extraction module (PFEM) is constructed to obtain the semantic information of the parts in the image using the low-level multi-scale features. Loss calculation is performed with the real part labels to provide auxiliary supervision for the part semantic information. Next, an instance mask confidence module (IMCM) is constructed to generate instance mask confidences consistent with the number of predicted targets. Finally, the obtained instance-semantic features are fused to generate the segmentation result of the target part.

[0074] The instance-semantic fusion segmentation method for part-level image segmentation of the present invention is specifically implemented according to the following steps:

[0075] Step 1: Build an instance-semantic fusion segmentation network based on FCOS. FCOS (Fully Convolutional One-Stage Object Detection, a single-stage object detection algorithm) is used to extract multi-scale convolutional features of the image and detect foreground objects, obtaining object categories and bounding box information.

[0076] Step 2: Build a part feature extraction module in the instance-semantic fusion segmentation network, and use it to perform a series of convolution, upsampling, and element-wise addition operations on the obtained multi-scale convolution features to extract the part semantic information in the image, that is, all possible target part feature maps;

[0077] Step 3: Calculate the loss between the obtained target part feature map and the real part label to provide auxiliary supervision for the extracted part semantic information;

[0078] Step 4: Build an instance mask confidence module and connect it in parallel to the object detection head. The convolution blocks in series generate the same number of instance mask confidences as the number of predicted objects.

[0079] In step 5, the target part feature map obtained in step 2 and the instance mask confidence obtained in step 4 are fused to obtain the part segmentation mask of all predicted targets.

[0080] Therefore, this paper combines the advantages of object detection and semantic segmentation to disclose an instance-semantic fusion segmentation method for part-level image segmentation. This method can not only locate and identify objects, but also segment the different parts of each object at the pixel level. This sophisticated segmentation capability can be fully applied in many fields in real life.

[0081] The present invention is further illustrated by the following experimental results.

[0082] 1. Experimental dataset.

[0083] The dataset used in the experiment is MVPOP. Figure 3 It is the prediction result of some images in MVPOP.

[0084] 2. Experimental content

[0085] The instance-semantic fusion segmentation network model is analyzed on the MVPOP dataset. From Tables 1, 2, and 3, it can be seen that the network model proposed in this invention has achieved excellent results in both target detection and target part segmentation.

[0086] Table 1 Test of segmentation network models on the MVPOP dataset

[0087] Backbone Epoch FPS mAP mIoU R50 12 20 0.69 0.77

[0088] Table 2 Target detection accuracy of each category

[0089] Bridge Car Bus Truck Airplane Carrier Cargoship Warship 0.48 0.85 0.82 0.78 0.48 0.76 0.66 0.65

[0090] Table 3 Segmentation accuracy of each category

[0091] Bridge Car Bus Truck Airplane Carrier Cargoship Warship 0.87 0.89 0.86 0.88 0.54 0.72 0.71 0.71 .

[0092] Example 2

[0093] like Figure 2 As shown, different from Example 1, step 1 of the instance-semantic fusion segmentation method for part-level image segmentation of the present invention is specifically implemented according to the following steps:

[0094] Step 1.1: Preprocess the input image, including resizing and normalization. The process is expressed as follows:

[0095]

[0096] Where I represents the input image, resize is the size scaling operation, μ is the mean of the image, σ is the standard deviation of the image, and I′ is the image after preprocessing.

[0097] Step 1.2: Pass the preprocessed image through the backbone network (i.e. Figure 2 The ResNet in

[15] is used for convolutional feature extraction; the backbone network is mainly composed of multiple residual blocks, which are used to extract semantic features at different levels. The process is expressed as follows:

[0098] {C1,C2,C3,C4,C5}=ResNet(I′) (2)

[0099] Among them, C1, C2, C3, C4, and C5 are the feature maps of each layer output by the backbone network. represents the feature map of layer i, d i is the number of channels;

[0100] Step 1.3, C3, C4 and C5 are passed through the feature pyramid network (i.e. Figure 2 The feature pyramid network introduces a top-down and horizontal connection structure to fuse feature maps at different levels to obtain multi-scale and semantically rich features. The process is expressed as follows:

[0101] P l =Conv3×3(Upsample(P l+1 )+Conv1×1(C l )),l=5,4,3 (3)

[0102] P m =Conv3×3(P m-1 ),m=6,7 (4)

[0103] Among them, Conv1×1 represents a convolution with a kernel size of 1×1, which is used to adjust the channel; Conv3×3 represents a convolution with a kernel size of 3×3, which is used to smooth the fused features; Upsample(·) is an upsampling operation, usually the nearest neighbor or bilinear interpolation. Finally, the multi-scale feature maps P3, P4, P5, P6 and P7 are obtained;

[0104] Step 1.4: Send P3, P4, P5, P6, and P7 to the target detection head to detect foreground targets in the image. The target detection head consists of a classification branch, a regression branch, and a centering branch. The classification branch generates a predicted category score, the regression branch generates target bounding box regression parameters, and the centering branch outputs a centering value.

[0105] Classification prediction is used to determine the category to which the target belongs. The process is expressed as:

[0106]

[0107] Among them, σ represents the sigmoid function, Conv clsIt is a number of convolutional layers used in the classification branch;

[0108] Bounding box regression is used to determine the location and size of the target bounding box. The process is expressed as:

[0109] t x,y =ReLU(Conv reg (P l (x,y)))=(l,t,r,b),l=3,4,5,6,7 (6)

[0110] Among them, ReLU represents RELU activation function, Conv reg are the convolutional layers used by the regression branch, l, t, r, b represent the distances from the center point to the left, top, right, and bottom of the bounding box;

[0111] The centrality prediction represents the distance between the target and the center point. The process is expressed as:

[0112] c x,y =σ(Conv ctr (P l (x,y))),l=3,4,5,6,7 (7)

[0113] Among them, σ represents the sigmoid function, Conv ctr are several convolutional layers used in the center branch.

[0114] Example 3

[0115] Combine Figure 2 , which is different from Example 2, in the example-semantic fusion segmentation method for part-level image segmentation of the present invention, step 2 is specifically implemented according to the following steps:

[0116] In step 2.1, P3, P4, and P5 are fused through convolution, upsampling, and element-wise addition to generate a high-resolution, semantically rich unified feature map. The process is expressed as follows:

[0117] F fusion =CBR(P3)+Up 2 (CBR(P4))+Up 4 (CBR(P5)) (8)

[0118] Among them, CBR represents Conv, BN and ReLU operations, Up 2 and Up 4 Respectively represent upsampling by 2 times and upsampling by 4 times, F fusion It is the feature map after the fusion of P3, P4 and P5;

[0119] Step 2.2, fuse the feature map F fusionAfter the concatenated convolution blocks and upsampling operations, the target part feature map representing the semantic information of the image part is obtained. The process is expressed as follows:

[0120] F part =Conv·(Up 2 CBR 4 )(F fusion ) (9)

[0121] Among them, CBR 4 Represents 4 consecutive Conv, BN and ReLU operations, and the size of the resulting part feature map T is (N, H / 4, W / 4), where N is the number of possible target parts, and H and W are the width and height of the original image.

[0122] Example 4

[0123] like Figure 2 As shown, different from Example 3, step 3 of the instance-semantic fusion segmentation method for part-level image segmentation of the present invention is specifically implemented according to the following steps:

[0124] In step 3.1, to ensure label integrity during the loss function calculation, the true labels of the image need to be preprocessed. Specifically, for the part mask not included, background pixels are used to fill the gaps. This operation not only ensures that all predicted positions have corresponding supervision signals, but also avoids the uncertainty caused by missing masks in the loss calculation.

[0125] In step 3.2, in order to solve the problem of small target area and imbalanced categories, an improved cross entropy loss function with weights and boundary guidance is used to make it more suitable for auxiliary supervision of part semantic information. The formula is expressed as:

[0126]

[0127] Among them, α c is the weight of different part categories, which is set to the inverse of the frequency of the category in the training set, that is:

[0128]

[0129] w i It is the position weight used for boundary enhancement, which can be used to control the importance of the boundary area by the boundary mask generated by the edge detector:

[0130]

[0131] Example 5

[0132] Combine Figure 2, which is different from Example 4, in step 4 of the instance-semantic fusion segmentation method for part-level image segmentation of the present invention: construct an instance mask confidence module (IMCM), (IMCM) is connected in parallel to the target detection head, and generates instance mask confidence feature maps with the same number as the predicted targets through the series of convolution blocks. The feature map size is (M, H / 8, W / 8, N), where M is the number of predicted targets, H and W are the width and height of the original image, and N is the number of possible target parts. The process is expressed as follows:

[0133] F conf =σ(Conv(CGR 4 (P l (x,y)))),l=3,4,5,6,7 (13)

[0134] Among them, CGR represents Conv, GN and ReLU operations, and σ represents the sigmoid activation function.

[0135] Example 6

[0136] Combine Figure 2 , which is different from Example 5, in step 5 of the instance-semantic fusion segmentation method for part-level image segmentation of the present invention, the following steps are specifically implemented:

[0137] Step 5.1, since F conf Yuan Figure 1 / 8 feature map, and F part Yuan Figure 1 / 4 feature map, so before fusion, we first conf Perform upsampling interpolation operation, and F part The feature map size is aligned, and the process is expressed as:

[0138] F conf '=inter 2 (F conf ) (14)

[0139] in, Represents a 2x bilinear interpolation operation, and the resulting F conf 'Size is (M,H / 4,W / 4,N), and F part consistent;

[0140] Step 5.2, and F part The multiplication and summation operation in the channel direction can be:

[0141]

[0142] F conf 'Provide the top instance information of the image, Fpart Provide the underlying part semantic information, and the two are fused to obtain the final target part segmentation result, namely Result.

Claims

1. An instance-semantic fusion segmentation method for part-level image segmentation, characterized in that: include: Step 1: Build an instance-semantic fusion segmentation network based on FCOS. FCOS is used to extract multi-scale convolutional features of the image and detect foreground objects. Step 2: Construct a part feature extraction module and use it to process the obtained multi-scale convolution features to extract the part semantic information in the image, that is, all possible target part feature maps; Step 3: Calculate the loss between the obtained target part feature map and the real part label to provide auxiliary supervision for the extracted part semantic information; Step 4: Build an instance mask confidence module and connect it in parallel to the object detection head. The convolution blocks in series generate the same number of instance mask confidences as the number of predicted objects. In step 5, the target part feature map obtained in step 2 and the instance mask confidence obtained in step 4 are fused to obtain the part segmentation mask of all predicted targets.

2. The instance-semantic fusion segmentation method for part-level image segmentation according to claim 1, characterized in that: Use it to process the obtained multi-scale convolution features.

3. The instance-semantic fusion segmentation method for part-level image segmentation according to claim 2, characterized in that: The step 1 is specifically implemented according to the following steps: Step 1.1: Preprocess the input image, including resizing and normalization. The process is expressed as follows: Where I represents the input image, resize is the size scaling operation, μ is the mean of the image, σ is the standard deviation of the image, and I′ is the image after preprocessing; Step 1.2: The preprocessed image is passed through the backbone network for convolutional feature extraction. The backbone network is mainly composed of multiple residual blocks, which are used to extract semantic features at different levels. The process is expressed as follows: {C1,C2,C3,C4,C5}=ResNet(I′) (2) Among them, C1, C2, C3, C4, and C5 are the feature maps of each layer output by the backbone network. represents the feature map of layer i, d i is the number of channels; Step 1.3: Pass C3, C4, and C5 through the feature pyramid network. The feature pyramid network introduces a top-down and horizontal connection structure to fuse feature maps at different levels, thereby obtaining multi-scale, semantically rich features. The process is expressed as follows: P l =Conv3×3(Upsample(P l+1 )+Conv1×1(C l )),l=5,4,3 (3) P m =Conv3×3(P m-1 ),m=6,7 (4) Among them, Conv1×1 represents a convolution with a convolution kernel size of 1×1, which is used to adjust the channel; Conv3×3 represents a convolution with a convolution kernel size of 3×3, which is used to smooth the fused features; Upsample(·) is an upsampling operation, usually the nearest neighbor or bilinear interpolation; finally, the P3, P4, P5, P6 and P7 multi-scale feature maps are obtained; Step 1.4: Send P3, P4, P5, P6, and P7 to the target detection head to detect the foreground target in the image; Classification prediction is used to determine the category to which the target belongs. The process is expressed as: Among them, σ represents the sigmoid function, Conv cls It is a number of convolutional layers used in the classification branch; Bounding box regression is used to determine the location and size of the target bounding box. The process is expressed as: t x,y =ReLU(Conv reg (P l (x,y)))=(l,t,r,b),l=3,4,5,6,7 (6) Among them, ReLU represents RELU activation function, Conv reg are the convolutional layers used by the regression branch, l, t, r, b represent the distances from the center point to the left, top, right, and bottom of the bounding box; The centrality prediction represents the distance between the target and the center point. The process is expressed as: c x,y =σ(Conv ctr (P l (x,y))),l=3,4,5,6,7 (7) Among them, σ represents the sigmoid function, Conv ctr are several convolutional layers used in the center branch.

4. The instance-semantic fusion segmentation method for part-level image segmentation according to claim 3, characterized in that: In step 1.4, the target detection head includes a classification branch, a regression branch, and a centerness branch. The classification branch generates a predicted category score, the regression branch generates a target bounding box regression parameter, and the centerness branch outputs a centerness value.

5. The instance-semantic fusion segmentation method for part-level image segmentation according to claim 4, characterized in that: The step 2 is specifically implemented according to the following steps: In step 2.1, P3, P4, and P5 are fused through convolution, upsampling, and element-wise addition to generate a high-resolution, semantically rich unified feature map. The process is expressed as follows: F fusion =CBR(P3)+Up 2 (CBR(P4))+Up 4 (CBR(P5)) (8) Among them, CBR represents Conv, BN and ReLU operations, Up 2 and Up 4 Respectively represent upsampling by 2 times and upsampling by 4 times, F fusion It is the feature map after the fusion of P3, P4 and P5; Step 2.2, fuse the feature map F fusion After the concatenated convolution blocks and upsampling operations, the target part feature map representing the semantic information of the image part is obtained. The process is expressed as follows: F part =Conv(Up 2 ·CBR 4 )(F fusion ) (9) Among them, CBR 4 Represents 4 consecutive Conv, BN and ReLU operations, and the size of the resulting part feature map T is (N, H / 4, W / 4), where N is the number of possible target parts, and H and W are the width and height of the original image.

6. The instance-semantic fusion segmentation method for part-level image segmentation according to claim 5, characterized in that: The step 3 is specifically implemented according to the following steps: Step 3.1, preprocess the true label of the image; In step 3.2, an improved cross entropy loss function with weights and boundary guidance is used to make it more suitable for auxiliary supervision of part semantic information. The formula is expressed as: Among them, α c is the weight of different part categories, which is set to the inverse of the frequency of the category in the training set, that is: w i It is the position weight used for boundary enhancement, which can be used to control the importance of the boundary area by the boundary mask generated by the edge detector:

7. The instance-semantic fusion segmentation method for part-level image segmentation according to claim 6, characterized in that: In step 4: construct an instance mask confidence module (IMCM), which is connected in parallel to the object detection head, and generates instance mask confidence feature maps with the same number of predicted objects through serial convolution blocks. The feature map size is (M, H / 8, W / 8, N), where M is the number of predicted objects, H and W are the width and height of the original image, and N is the number of possible object parts. The process is expressed as: F conf =σ(Conv(CGR 4 (P l (x,y)))),l=3,4,5,6,7 (13) Among them, CGR represents Conv, GN and ReLU operations, and σ represents the sigmoid activation function.

8. The instance-semantic fusion segmentation method for part-level image segmentation according to claim 7, characterized in that: The step 5 is specifically implemented according to the following steps: Step 5.1, before fusion, first conf Perform upsampling interpolation operation, and F part Align the feature map sizes; The process is expressed as: F conf '=inter 2 (F conf ) (14) Among them, inter 2 Represents a 2x bilinear interpolation operation, and the resulting F conf 'Size is (M,H / 4,W / 4,N), and F part consistent; Step 5.2, for F conf ' and F part The multiplication and summation operation in the channel direction can be: F conf 'Provide the top instance information of the image, F part Providing the semantic information of the underlying parts, the two are fused to obtain the final target part segmentation result.

Images