Target instance part segmentation method for part-level image segmentation

Abstract

The invention discloses a target instance part segmentation method for part-level image segmentation, and the method comprises the steps: carrying out the preprocessing of an input image, carrying out the foreground target detection through a Faster R-CNN network, and obtaining candidate region features generated by a region proposal network; and then, constructing a target part segmentation branch to obtain a part segmentation mask, constraining the part segmentation mask by adopting part perception segmentation loss, and improving the segmentation precision of different parts of the target by combining the part perception segmentation loss with a plurality of pixel-level loss functions. And finally, fully training the target detection branch by using a progressive training strategy, and then training the target detection branch and the target part segmentation branch at the same time, thereby improving the network detection and segmentation quality. According to the invention, the problem that the existing image segmentation method is insufficient in target part recognition is solved.

Description

Technical Field

[0001] The present invention belongs to the field of information processing technology, and in particular relates to a target instance part segmentation method for part-level image segmentation. Background Art

[0002] With the rapid development of science and technology, computer vision is gradually penetrating various fields of industrial production, social life, and scientific research. Image segmentation, as a fundamental task in computer vision, has garnered widespread attention due to its crucial role in image understanding. Image segmentation distinguishes target objects from the background in an image, allowing for independent analysis and processing of each region. This technology has widespread application in many fields, such as medical image analysis, autonomous driving, and security monitoring. With the advancement of deep learning technology, neural network-based image segmentation methods have also made significant progress, significantly improving the accuracy and efficiency of image segmentation.

[0003] Compared with traditional image segmentation, target instance part segmentation introduces the requirement of target part recognition. On the basis of segmenting the overall target, it further segments the various parts of the target. Specifically, target instance part segmentation has two requirements: (1) Instance detection: treat the entire target in the scene as an instance, accurately locate and identify each instance. (2) Part segmentation: On the basis of identifying the overall target, further segment the various parts of the target. Therefore, in target instance part segmentation, the challenge faced by the algorithm is not only to accurately identify and locate the foreground target in the image or video stream. The more complex and sophisticated requirement is to be able to further penetrate into the internal structure of the target object and accurately identify and segment the various parts that constitute these targets, such as the fuselage, wings, and engines of an aircraft, or the deck, chord, and command room of a ship.

[0004] At present, the work on target instance part segmentation can be introduced from two aspects: semantic part segmentation and instance segmentation.

[0005] Semantic part segmentation involves segmenting individual parts within an object, rather than the entire object in a scene, as is done with semantic segmentation. Wang J et al. proposed a combined model combining shape and appearance features to address the problem of semantic part segmentation in animals. Xia F et al. improved human image processing capabilities by jointly addressing two complementary tasks in computer vision: human pose estimation and semantic part segmentation. Rewatbowornwong P et al. leveraged the generative and discriminative capabilities of GANs to accurately segment the semantic parts of objects in an image using only one or a small number of labeled samples. Although research in the field of semantic part segmentation has achieved considerable success, the following limitations remain: 1. Current semantic part segmentation algorithms primarily segment different body parts of humans and animals and have not been extended to other objects. 2. Semantic part segmentation algorithms are inherently based on semantic segmentation and cannot distinguish between the same parts belonging to different instances. 3. Semantic part segmentation algorithms cannot identify and localize the entire object. Therefore, developing an algorithm that can identify different object instances and accurately segment different parts of an object is an ideal goal for object part recognition.

[0006] Instance segmentation is a high-level task in computer vision that combines the characteristics of object detection and semantic segmentation. Existing instance segmentation methods can be divided into three categories: region-based methods, center-based methods, and query-based methods. Region-based methods predict bounding boxes through object detection and then perform segmentation within each bounding box. Representative methods include Mask R-CNN. Center-based methods, based on the idea of single-stage object detection, can perform segmentation and detection in parallel. Representative methods include YOLACT and SOLO. Query-based methods leverage a powerful and general attention mechanism, using queries to represent objects of interest, and simultaneously perform classification, regression, and mask prediction. Representative methods include QueryInst and FastInst. While existing instance segmentation methods have demonstrated high maturity and accuracy in many scenarios, performance improvements are gradually reaching a bottleneck due to software and hardware limitations. With the growing demand for practical applications, accurately obtaining object part information is becoming increasingly important. Therefore, object instance part segmentation methods will play an increasingly important role in the future of image processing and computer vision. Summary of the Invention

[0007] The purpose of the present invention is to provide a target instance part segmentation method for part-level image segmentation, so as to solve the problem that existing image segmentation methods are insufficient in identifying target parts.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is: a target instance part segmentation method for part-level image segmentation, comprising:

[0009] Step 1: Build an instance segmentation network based on Faster R-CNN. Faster R-CNN is used for foreground object detection and provides candidate region features generated by the region proposal network in Faster R-CNN.

[0010] Step 2: construct a target part segmentation branch of the target instance part segmentation network, which is used to accept the candidate region features obtained in step 1 and generate a part segmentation mask corresponding to each candidate region feature;

[0011] Step 3: Design a part-aware segmentation loss and use it to constrain the part segmentation mask generated in step 2 to guide the training process of the target part segmentation branch;

[0012] Step 4: First, train the object detection branch in Faster R-CNN, and then train it simultaneously with the object segmentation branch to improve the network detection and segmentation quality.

[0013] The technical solution of the present invention also has the following characteristics:

[0014] Step 1 is implemented as follows:

[0015] Step 1.1: Normalize the input image using mean-standard deviation. The process is expressed as:

[0016]

[0017] In formula (1), I represents the original image, μ is the mean of each channel of the image, σ is the standard deviation of each channel, and I′ is the image after normalization;

[0018] In step 1.2, I′ passes through the residual network ResNet, which gradually converts the original input data into a high-dimensional feature representation, and obtains three layers of feature maps C3, C4 and C5 with decreasing resolutions.

[0019] In step 1.3, C3, C4, and C5 are fed into the feature pyramid network FPN to fuse the semantic information of the high-level feature maps with the spatial information of the low-level feature maps to generate feature maps P3, P4, P5, P6, and P7 with rich multi-scale information. The process is expressed as follows:

[0020] P i =W i C i +Upsample(P i+1 ) (2)

[0021] Formula (2) Where, W i is a 1×1 convolution, used to adjust the number of channels; Upsample(·) indicates 2x upsampling; the resulting P3 resolution is 1 / 4 of the original image, the P4 resolution is 1 / 8 of the original image, the P5 resolution is 1 / 16 of the original image, the P6 resolution is 1 / 32 of the original image, and the P7 resolution is 1 / 64 of the original image;

[0022] Step 1.4: Input the FPN features P3, P4, P5, P6, and P7 into the region proposal network RPN to obtain candidate regions where the target area may exist.

[0023] Step 1.5, by using the location of the candidate region Proposals to crop the corresponding area on the feature maps P3, P4, P5, P6 and P7 obtained in step 1.3, the convolution feature map of the candidate region is obtained, and the pooling operation is used to unify it into a fixed size. The process is expressed as:

[0024] F = RoIAlign(Crop(P i )) (3)

[0025] In formula (3), Crop is to crop the corresponding feature area, and RoIAlign is a pooling technology that unifies the cropped area into a 7×7 size;

[0026] In step 1.6, F detects the foreground objects in the image through the object detection branch. This branch contains two parallel fully connected networks that accept the feature maps F of all candidate regions. One is combined with a classifier to classify the candidate regions, and the other further optimizes the coordinates of the candidate regions. The process is expressed as follows:

[0027] cls=Softmax(FC3(FC2(FC1(Flatten(F))))) (4)

[0028] reg=FC4(FC2(FC1(Flatten(F)))) (5)

[0029] Formula (4) and Formula (5) In which, Flatten is used to flatten the feature map, FC represents the fully connected layer; cls is the classification result, and the original output logits are converted into probability distribution through Softmax; reg is the regression branch, which determines the bounding box of the target.

[0030] In step 1.1, ResNet is composed of a combination of convolutional layers, pooling layers, and activation functions.

[0031] In step 1.1, the resolution of C3 is 1 / 4 of the original image, the resolution of C4 is 1 / 8 of the original image, and the resolution of C5 is 1 / 16 of the original image.

[0032] Step 2 is implemented as follows:

[0033] Step 2.1: Use the candidate region feature Proposals obtained in step 1.4 to crop the corresponding region on the feature maps P3, P4, P5, P6 and P7 obtained in step 1.3 to obtain the convolution feature map of the candidate region, and use the pooling operation to unify it into a fixed size. The process is expressed as:

[0034] F′=RoIAlign(Crop(P i )) (6)

[0035] In Formula 6, Crop is the cropping of the corresponding feature region, and RoIAlign is a pooling technique that unifies the cropped region into a 14×14 size to obtain F′;

[0036] In step 2.2, F′ is fed into the context information extraction module to further extract context information from the feature map. The process is expressed as follows:

[0037]

[0038] In formula (7), Conv1 is a convolution with a kernel size of 1×1, Conv3 is a convolution with a kernel size of 3×3, Upsample represents an upsampling operation, Pooling represents a pooling operation, and Pglobal represents a global average pooling operation. Finally, all feature maps y i All of them are concatenated and compressed using 1×1 convolution to obtain F″;

[0039] In step 2.3, after context information extraction, F″ passes through 4 series convolution blocks and transposed convolution, and finally the number of feature map channels is compressed to 1 to obtain the part segmentation result of the feature area. The process is expressed as follows:

[0040] m=ReLu(Conv3(F″))×4 (8)

[0041]

[0042] In formula (8) and formula (9), Conv3 is a convolution with a kernel size of 3×3, ReLu is an activation function, ConvTrans2 is a transposed convolution with a kernel size of 2×2, and Conv1 is a convolution with a kernel size of 1×1. Finally, a 28×28 part segmentation mask is obtained.

[0043] Step 3 is implemented as follows:

[0044] In step 3.1, the category-weighted cross entropy loss WCE is introduced. The WCE loss function assigns different weights to different categories, which is expressed as:

[0045]

[0046] In formula (10), y i is the one-hot encoding of the true category, y i is the network prediction result, Category c i The weight of , the commonly used formula is:

[0047]

[0048] In formula (11), freq(c) is the proportion of pixels of category c in the dataset, and categories with fewer pixels have higher weights;

[0049] In step 3.2, the pixels of the target part should have local consistency, that is, the prediction probabilities within the same part should be similar. To this end, the similarity loss PCL between local pixels is introduced, and the formula is as follows:

[0050]

[0051] In formula (12), p represents the location of a target, Ω p Represents the set of all pixels in part p, N p is the number of pixels in the part p, y i and y j is the predicted distribution of pixel i,j;

[0052] Step 3.3: Use boundary smoothing loss (BSL) to improve the segmentation boundary quality of the target part and avoid drastic fluctuations in the predicted values of boundary pixels. The formula is as follows:

[0053]

[0054] In formula (13), Used to calculate the gradient at pixel i;

[0055] In step 3.4, the overall part-aware segmentation loss PSL is expressed as:

[0056] L PSL =αL WCE +βL PCL +γL BSL (14)

[0057] In formula (14), α, β and γ are equilibrium parameters, LWCE To solve the imbalance of categories, L PCL To promote internal consistency of the site, L BSL Improve boundary quality.

[0058] Step 4 includes:

[0059] In step 4.1, in the first 6 epochs, freeze the target segmentation branch and focus on the training of the target detection branch, so that the network can accurately identify target instances and perform precise boundary positioning.

[0060] Step 4 includes:

[0061] In step 4.2, after the 7th Epoch, the object detection branch is fully trained, unfrozen and the object segmentation branch is introduced to optimize the detection and segmentation tasks simultaneously.

[0062] The beneficial effects of the present invention are:

[0063] (1) The present invention proposes a target instance part segmentation network, which is characterized by being able to achieve end-to-end target instance part segmentation. The network achieves fine target part segmentation within the detected target frame by connecting a target part segmentation branch in parallel with the target detection branch. This design enables the model to further delve into the internal structure of the target object on the basis of identifying the entire target. This is of great significance in practical applications. For example, in the field of autonomous driving, it can achieve accurate identification of various parts of the vehicle, thereby improving the safety and reliability of the autonomous driving system.

[0064] (2) The present invention greatly improves the accuracy of part segmentation. The present invention uses part-aware segmentation loss to constrain the part segmentation mask. The class-weighted cross entropy loss (WCE) effectively solves the class imbalance problem and prevents small parts from being ignored. The local pixel similarity loss (PCL) promotes internal consistency of the part and reduces the fragmentation of the segmentation. The boundary smoothing loss (BSL) improves the quality of the part segmentation boundary and reduces jagged edges.

[0065] (3) The present invention provides a good model training strategy. A progressive training strategy is used, where the object detection branch is first fully trained and then trained simultaneously with the object segmentation branch. Through this progressive training strategy, the network can first learn global object detection capabilities and then further optimize the fine-grained object segmentation task, thereby improving the overall detection and segmentation quality, and making the final model more generalizable and robust in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of the overall design of the target instance part segmentation method for part-level image segmentation of the present invention.

[0067] Figure 2 It is a schematic diagram of the framework of the target instance part segmentation method for part-level image segmentation of the present invention.

[0068] Figure 3 This is the overall training process of the target instance part segmentation method for part-level image segmentation of the present invention.

[0069] Figure 4 It is the segmentation result of the target instance part segmentation method for part-level image segmentation of the present invention on the MVPOP dataset. DETAILED DESCRIPTION

[0070] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] Example 1

[0072] The purpose of this invention is to address the shortcomings of existing image segmentation methods in identifying target parts and propose a new target instance part segmentation method to achieve precise instance detection and part segmentation results. This method uses a target part segmentation branch to segment the target parts within the detected target frame. Since the present invention performs part segmentation within the target detection frame, the segmentation accuracy is very high. Figure 1 As shown, the input image is first preprocessed, and the Faster R-CNN network is used to detect foreground objects. Meanwhile, candidate region features are generated by the region proposal network. Next, a target part segmentation branch is constructed to obtain the part segmentation mask. This is constrained using a part-aware segmentation loss. This part-aware segmentation loss combines multiple pixel-level loss functions to improve the segmentation accuracy of different target parts. Finally, a progressive training strategy is used to fully train the target detection branch first, and then train it simultaneously with the target part segmentation branch to improve network detection and segmentation quality.

[0073] The target instance part segmentation method for part-level image segmentation of the present invention is specifically implemented according to the following steps:

[0074] Step 1: Build an object instance segmentation network based on Faster R-CNN. Faster R-CNN (Faster R-CNN is a two-stage object detection algorithm) is used for foreground object detection and provides candidate region features generated by the region proposal network in Faster R-CNN.

[0075] Step 2: construct a target part segmentation branch of the target instance part segmentation network, which is used to accept the candidate region features obtained in step 1 and generate a part segmentation mask corresponding to each candidate region feature;

[0076] Step 3: Design a part-aware segmentation loss and use it to constrain the part segmentation mask generated in step 2 to guide the training process of the target part segmentation branch;

[0077] Step 4: First, train the object detection branch in Faster R-CNN, and then train it simultaneously with the object segmentation branch to improve the network detection and segmentation quality.

[0078] Reference Figure 3 The overall training process of the target instance part segmentation network for part-level instance segmentation of the present invention is as follows:

[0079] Step 1: Image data acquisition.

[0080] Collect image data from multiple height perspectives from remote sensing images, images taken by drones, and images taken by ground handheld devices.

[0081] Step 2: Image preprocessing.

[0082] The collected images are filtered, cropped, and filled as necessary. After preprocessing, the size of all images in the dataset is limited to a specific range, while ensuring that the position of the target object in the image is completely random.

[0083] Step 3: Data labeling.

[0084] The target in the image is marked with a horizontal rectangular frame, and each part of the target is marked with a polygon.

[0085] Step 4: Generate a high-quality multi-view target part dataset.

[0086] The labeled images are processed into annotation files and divided into training sets, validation sets and test sets.

[0087] Step 5: Load the dataset.

[0088] Through the loop function, a batch of images and annotation data are loaded each time and input into the network model.

[0089] Step 6: Data processing.

[0090] Normalize the loaded image data and perform necessary padding operations.

[0091] Step 7: Model training.

[0092] According to the progressive training strategy, the target detection branch is first fully trained, and then trained simultaneously with the target part segmentation branch. The model is verified during the training process to preserve the optimal model results.

[0093] The effects of the present invention can be further illustrated by the following experimental results.

[0094] 1. Experimental dataset.

[0095] The dataset used in the experiment is MVPOP. Figure 4 It is the prediction result of some images in MVPOP.

[0096] 2. Experimental content.

[0097] The target instance part segmentation network model is analyzed on the MVPOP dataset. From Tables 1, 2, and 3, it can be seen that the model proposed in this invention has achieved good results in both target detection and target part segmentation.

[0098] Table 1 Test of the model proposed in this invention on the MVPOP dataset

[0099] Backbone Epoch mAP mIoU R50 12 0.70 0.81

[0100] Table 2 APs of various categories

[0101] Bridge Car Bus Truck Airplane Carrier Cargoship Warship 0.46 0.86 0.79 0.78 0.76 0.73 0.63 0.61

[0102] Table 3 mIoU for each category

[0103] Bridge Car Bus Truck Airplane Carrier Cargoship Warship 0.92 0.90 0.84 0.90 0.73 0.80 0.76 0.73 .

[0104] Example 2

[0105] like Figure 2 As shown, different from Example 1, step 1 of the target instance part segmentation method for part-level image segmentation of the present invention is specifically implemented according to the following steps:

[0106] Step 1.1: Normalize the input image using mean-standard deviation. The process is expressed as:

[0107]

[0108] In formula (1), I represents the original image, μ is the mean of each channel of the image, σ is the standard deviation of each channel, and I′ is the image after normalization;

[0109] In step 1.2, I′ passes through the residual network ResNet. ResNet gradually converts the original input data into a high-dimensional feature representation through a combination of a series of convolutional layers, pooling layers, and activation functions, obtaining three layers of feature maps C3, C4, and C5 with decreasing resolutions. The resolution of C3 is 1 / 4 of the original image, the resolution of C4 is 1 / 8 of the original image, and the resolution of C5 is 1 / 16 of the original image.

[0110] In step 1.3, C3, C4, and C5 are fed into the feature pyramid network FPN to fuse the semantic information of the high-level feature maps with the spatial information of the low-level feature maps to generate feature maps P3, P4, P5, P6, and P7 with rich multi-scale information. The process is expressed as follows:

[0111] P i =W i C i +Upsample(P i+1 ) (2)

[0112] Formula (2) Where, W i is a 1×1 convolution, used to adjust the number of channels. Upsample(·) indicates 2x upsampling; the resulting P3 resolution is 1 / 4 of the original image, the P4 resolution is 1 / 8 of the original image, the P5 resolution is 1 / 16 of the original image, the P6 resolution is 1 / 32 of the original image, and the P7 resolution is 1 / 64 of the original image.

[0113] Step 1.4: Input the FPN features P3, P4, P5, P6, and P7 into the region proposal network RPN to obtain candidate regions where the target area may exist.

[0114] Step 1.5, by using the location of the candidate region Proposals to crop the corresponding area on the feature maps P3, P4, P5, P6 and P7 obtained in step 1.3, the convolution feature map of the candidate region is obtained, and the pooling operation is used to unify it into a fixed size. The process is expressed as:

[0115] F = RoIAlign(Crop(P i )) (3)

[0116] In formula (3), Crop is to crop the corresponding feature area, and RoIAlign is a pooling technology that unifies the cropped area into a 7×7 size;

[0117] In step 1.6, F detects the foreground objects in the image through the object detection branch. This branch contains two parallel fully connected networks that accept the feature maps F of all candidate regions. One is combined with a classifier to classify the candidate regions, and the other further optimizes the coordinates of the candidate regions. The process is expressed as follows:

[0118] cls=Softmax(FC3(FC2(FC1(Flatten(F))))) (4)

[0119] reg=FC4(FC2(FC1(Flatten(F)))) (5)

[0120] Formula (4) and Formula (5) In which, Flatten is used to flatten the feature map, FC represents the fully connected layer; cls is the classification result, and the original output logits are converted into probability distribution through Softmax; reg is the regression branch, which determines the bounding box of the target.

[0121] Example 3

[0122] Different from Example 2, step 2 of the target instance part segmentation method for part-level image segmentation of the present invention is specifically implemented according to the following steps:

[0123] Step 2.1: Use the candidate region feature Proposals obtained in step 1.4 to crop the corresponding region on the feature maps P3, P4, P5, P6 and P7 obtained in step 1.3 to obtain the convolution feature map of the candidate region, and use the pooling operation to unify it into a fixed size. The process is expressed as:

[0124] F′=RoIAlign(Crop(P i )) (6)

[0125] In Formula 6, Crop is the cropping of the corresponding feature region, and RoIAlign is a pooling technique that unifies the cropped region into a 14×14 size to obtain F′;

[0126] In step 2.2, F′ is fed into the context information extraction module to further extract context information from the feature map. The process is expressed as follows:

[0127]

[0128] In formula (7), Conv1 is a convolution with a kernel size of 1×1, Conv3 is a convolution with a kernel size of 3×3, Upsample represents an upsampling operation, Pooling represents a pooling operation, and Pglobal represents a global average pooling operation. Finally, all feature maps y i All of them are concatenated and compressed using 1×1 convolution to obtain F″;

[0129] In step 2.3, after context information extraction, F″ passes through 4 series convolution blocks and transposed convolution, and finally the number of feature map channels is compressed to 1 to obtain the part segmentation result of the feature area. The process is expressed as follows:

[0130] m=ReLu(Conv3(F″))×4 (8)

[0131]

[0132] In formula (8) and formula (9), Conv3 is a convolution with a kernel size of 3×3, ReLu is an activation function, ConvTrans2 is a transposed convolution with a kernel size of 2×2, and Conv1 is a convolution with a kernel size of 1×1. Finally, a 28×28 part segmentation mask is obtained.

[0133] Example 4

[0134] like Figure 2 As shown, different from Example 3, step 3 of the target instance part segmentation method for part-level image segmentation of the present invention is specifically implemented according to the following steps:

[0135] In step 3.1, in order to solve the problem of class imbalance and make small part segmentation more stable, the class weighted cross entropy loss WCE is introduced. The WCE loss function assigns different weights to different classes, which is expressed as:

[0136]

[0137] In formula (10), y i is the one-hot encoding of the true category, y i is the network prediction result, Category c i The weight of , the commonly used formula is:

[0138]

[0139] In formula (11), freq(c) is the proportion of pixels of category c in the dataset, and categories with fewer pixels have higher weights;

[0140] In step 3.2, the pixels of the target part should have local consistency, that is, the prediction probabilities within the same part should be similar. To this end, the similarity loss PCL between local pixels is introduced, and the formula is as follows:

[0141]

[0142] In formula (12), p represents the location of a target, Ω p Represents the set of all pixels in part p, N p is the number of pixels in the part p, y i and y j is the predicted distribution of pixel i, j; through PCL loss, the predicted values of pixels in the same part are promoted to be consistent, reducing the fragmentation phenomenon in the prediction.

[0143] Step 3.3: Use boundary smoothing loss (BSL) to improve the segmentation boundary quality of the target part and avoid drastic fluctuations in the predicted values of boundary pixels. The formula is as follows:

[0144]

[0145] In formula (13), Used to calculate the gradient at pixel i;

[0146] In step 3.4, the overall part-aware segmentation loss PSL is expressed as:

[0147] L PSL =αL WCE +βL PCL +γL BSL (14)

[0148] Among them, α, β and γ are equilibrium parameters, L WCE To solve the imbalance of categories, L PCL To promote internal consistency of the site, L BSL Improve boundary quality.

[0149] Example 5

[0150] Different from Example 4, step 4 of the target instance part segmentation method for part-level image segmentation of the present invention includes: step 4.1, in the first 6 epochs, freezing the target part segmentation branch, focusing on the training of the target detection branch, so that the network can accurately identify the target instance and perform precise boundary positioning; the core goal of this stage is to enable the network to learn the global features of the target and improve the regression accuracy and classification performance of the target box.

[0151] Example 6

[0152] Different from Example 5, in Example 6, step 4 of a target instance part segmentation method for part-level image segmentation of the present invention includes: step 4.2, after the 7th Epoch, the target detection branch is fully trained, and the target part segmentation branch is unfrozen and introduced to optimize the detection and segmentation tasks simultaneously; at this time, the network not only needs to learn the global features of the target instance, but also needs to capture the fine-grained structural information inside the target to achieve accurate target part segmentation.

Claims

1. A target instance part segmentation method for part-level image segmentation, characterized in that: include: Step 1: Build an instance segmentation network based on Faster R-CNN. Faster R-CNN is used for foreground object detection and provides candidate region features generated by the region proposal network in Faster R-CNN. Step 2: construct a target part segmentation branch of the target instance part segmentation network, which is used to accept the candidate region features obtained in step 1 and generate a part segmentation mask corresponding to each candidate region feature; Step 3: Design a part-aware segmentation loss and use it to constrain the part segmentation mask generated in step 2 to guide the training process of the target part segmentation branch; Step 4: First, train the object detection branch in Faster R-CNN, and then train it simultaneously with the object segmentation branch to improve the network detection and segmentation quality.

2. The target instance part segmentation method for part-level image segmentation according to claim 1, characterized in that The step 1 is specifically implemented according to the following steps: Step 1.1: Normalize the input image using mean-standard deviation. The process is expressed as: In formula (1), I represents the original image, μ is the mean of each channel of the image, σ is the standard deviation of each channel, and I′ is the image after normalization; In step 1.2, I′ passes through the residual network ResNet, which gradually converts the original input data into a high-dimensional feature representation, and obtains three layers of feature maps C3, C4 and C5 with decreasing resolutions. In step 1.3, C3, C4, and C5 are fed into the feature pyramid network FPN to fuse the semantic information of the high-level feature maps with the spatial information of the low-level feature maps to generate feature maps P3, P4, P5, P6, and P7 with rich multi-scale information. The process is expressed as follows: P i =W i C i +Upsample(P i+1 ) (2) Formula (2) Where, W i It is a 1×1 convolution, which is used to adjust the number of channels; Upsample() means 2x upsampling; the resulting P3 resolution is 1 / 4 of the original image, the P4 resolution is 1 / 8 of the original image, the P5 resolution is 1 / 16 of the original image, the P6 resolution is 1 / 32 of the original image, and the P7 resolution is 1 / 64 of the original image; Step 1.4: Input the FPN features P3, P4, P5, P6, and P7 into the region proposal network RPN to obtain candidate regions where the target area may exist. Step 1.5, by using the location of the candidate region Proposals to crop the corresponding area on the feature maps P3, P4, P5, P6 and P7 obtained in step 1.3, the convolution feature map of the candidate region is obtained, and the pooling operation is used to unify it into a fixed size. The process is expressed as: F=RoIAlign(Crop(P i )) (3) In formula (3), Crop is to crop the corresponding feature area, and RoIAlign is a pooling technology that unifies the cropped area into a 7×7 size; In step 1.6, F detects the foreground objects in the image through the object detection branch. This branch contains two parallel fully connected networks that accept the feature maps F of all candidate regions. One is combined with a classifier to classify the candidate regions, and the other further optimizes the coordinates of the candidate regions. The process is expressed as follows: cls=Softmax(FC3(FC2(FC1(Flatten(F))))) (4) reg=FC4(FC2(FC1(Flatten(F)))) (5) In formula (4) and formula (5), Flatten is used to flatten the feature map, FC represents the fully connected layer; cls is the classification result, and the original output logits are converted into probability distribution through Softmax; reg is the regression branch, which determines the bounding box of the target.

3. The target instance part segmentation method for part-level image segmentation according to claim 2, characterized in that In step 1.1, ResNet is composed of a combination of convolutional layers, pooling layers, and activation functions.

4. The object instance part segmentation method for part-level image segmentation according to claim 3, characterized in that In step 1.1, the resolution of C3 is 1 / 4 of the original image, the resolution of C4 is 1 / 8 of the original image, and the resolution of C5 is 1 / 16 of the original image.

5. The method for object instance part segmentation for part-level image segmentation according to claim 4, wherein: The step 2 is specifically implemented according to the following steps: Step 2.1: Use the candidate region feature Proposals obtained in step 1.4 to crop the corresponding region on the feature maps P3, P4, P5, P6 and P7 obtained in step 1.3 to obtain the convolution feature map of the candidate region, and use the pooling operation to unify it into a fixed size. The process is expressed as: F′=RoIAlign(Crop(P i )) (6) In Formula 6, Crop is the cropping of the corresponding feature region, and RoIAlign is a pooling technique that unifies the cropped region into a 14×14 size to obtain F′; In step 2.2, F′ is fed into the context information extraction module to further extract context information from the feature map. The process is expressed as follows: In formula (7), Conv1 is a convolution with a kernel size of 1×1, Conv3 is a convolution with a kernel size of 3×3, Upsample represents an upsampling operation, Pooling represents a pooling operation, and Pglobal represents a global average pooling operation. Finally, all feature maps y i All of them are concatenated and compressed using 1×1 convolution to obtain F″; In step 2.3, after context information extraction, F″ passes through 4 series convolution blocks and transposed convolution, and finally the number of feature map channels is compressed to 1 to obtain the part segmentation result of the feature area. The process is expressed as follows: m=ReLu(Conv3(F″))×4 (8) In formula (8) and formula (9), Conv3 is a convolution with a kernel size of 3×3, ReLu is an activation function, ConvTrans2 is a transposed convolution with a kernel size of 2×2, and Conv1 is a convolution with a kernel size of 1×1. Finally, a 28×28 part segmentation mask is obtained.

6. The method for object instance part segmentation for part-level image segmentation according to claim 5, characterized in that: The step 3 is specifically implemented according to the following steps: In step 3.1, the category-weighted cross entropy loss WCE is introduced. The WCE loss function assigns different weights to different categories, which is expressed as: In formula (10), y i is the one-hot encoding of the true category, y i is the network prediction result, Category c i The weight of , the commonly used formula is: In formula (11), freq(c) is the proportion of pixels of category c in the dataset, and categories with fewer pixels have higher weights; In step 3.2, the pixels of the target part should have local consistency, that is, the prediction probabilities within the same part should be similar. To this end, the similarity loss PCL between local pixels is introduced, and the formula is as follows: In formula (12), p represents the location of a target, Ω p Represents the set of all pixels in part p, N p is the number of pixels in the part p, y i and y j is the predicted distribution of pixel i,j; Step 3.3: Use boundary smoothing loss (BSL) to improve the segmentation boundary quality of the target part and avoid drastic fluctuations in the predicted values of boundary pixels. The formula is as follows: In formula (13), Used to calculate the gradient at pixel i; In step 3.4, the overall part-aware segmentation loss PSL is expressed as: L PSL =αL WCE +βL PCL +γL BSL (14) Among them, α, β and γ are equilibrium parameters, L WCE To solve the imbalance of categories, L PCL To promote internal consistency of the site, L BSL Improve boundary quality.

7. The object instance part segmentation method for part-level image segmentation according to claim 6, characterized in that: The step 4 comprises: In step 4.1, in the first 6 epochs, freeze the target segmentation branch and focus on the training of the target detection branch, so that the network can accurately identify target instances and perform precise boundary positioning.

8. The object instance part segmentation method for part-level image segmentation according to claim 7, characterized in that: The step 4 comprises: In step 4.2, after the 7th Epoch, the object detection branch is fully trained, unfrozen and the object part segmentation branch is introduced to optimize the detection and segmentation tasks simultaneously.

Images