Contour-based Image Instance Segmentation Method and System

Abstract

The present invention relates to a contour-based image instance segmentation method and system. The method comprises inputting a target image into a lightweight feature extraction network to obtain a downsampled feature map, a center point, and the offset of contour vertices relative to the center point of the target instance in the image; calculating an initial contour based on the offset of the center point and the contour vertex relative to the center point; performing global feature fusion on the downsampled feature map of the target instance and the initial contour to obtain a rough contour; performing a first contour deformation on the rough contour to obtain a preliminary instance contour, and performing a second contour deformation on the preliminary instance contour to obtain a final instance contour. The present invention abandons the traditional method of manually designing an initial contour and instead learns an initial contour that is closer to the true value based on network learning. While maintaining the algorithm segmentation accuracy, it has a smaller number of parameters and computational complexity, and has a faster inference speed. It can be applied to instance segmentation tasks in real-world scenarios using edge platforms.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a contour-based image instance segmentation method and system. Background Art

[0002] Current state-of-the-art instance segmentation algorithms are based on deep learning techniques, typically using convolutional neural networks (CNNs) to extract features. These models have many layers and a large number of parameters, requiring extensive computational resources for training and inference, as well as large amounts of memory to store intermediate parameters. Consequently, they require significant computational and storage resources. For example, the Mask R-CNN model has 17 million parameters, requiring hundreds or even thousands of megabytes of computational power to process a single high-definition image, requiring extensive GPU acceleration. Such algorithms cannot be run directly on edge devices and require optimization. However, edge devices are limited by computational resources and energy consumption. Deploying instance segmentation models on edge platforms faces the following challenges: First, computational overhead: Most deep learning models consume significant computational resources, such as GPUs and TPUs. However, edge devices typically rely on low-power CPUs and FPGAs. Therefore, lightweight model architectures are necessary to reduce computational overhead. Second, memory usage: Deep learning models require significant memory to store weights and intermediate computational results. Given the limited memory available on edge devices, compression algorithms are necessary to reduce memory usage. Third, the problem of inference speed: In order to meet real-time requirements on resource-constrained edge platforms in real scenarios, it is necessary to reduce the inference time of the model as much as possible.

[0003] To address these challenges, research on instance segmentation algorithm optimization for edge intelligence is of paramount importance. On the one hand, algorithm optimization can reduce the model's computational complexity and memory usage through the use of lightweight models, quantization, pruning, and mixed precision, thereby improving real-time performance. On the other hand, deploying instance segmentation algorithms on edge platforms allows devices to quickly respond to user needs locally, avoiding data transmission and processing delays and thus improving the user experience. Furthermore, research on instance segmentation algorithm optimization based on edge intelligence can also help promote the integration of artificial intelligence and the Internet of Things. The integration of edge computing and artificial intelligence is a trend in the future intelligent society. Optimizing instance segmentation algorithms can provide technical support for practical scenarios such as autonomous driving, smart healthcare, and industrial robotics.

[0004] Currently, mainstream instance segmentation algorithms use two main annotation formats for representing instances: grids and contours. A grid mask is a pixel-based binary mask, labeling each pixel as foreground or background. Contour representations, on the other hand, represent a set of consecutive pixel coordinates representing the instance boundary. Grid masks can better describe the geometric structure and internal information of an object, but may require more computational resources. Contour representations are relatively lightweight, but fail to capture internal details. In some scenarios, such as trees and grass in natural images, contour representations are more suitable for capturing the external shape and texture of an object. Currently, the Deep Snake algorithm is widely used for real-time instance segmentation. This algorithm gradually optimizes an initial contour to the object boundary for object segmentation. Specifically, the object contour is used as input and the offset of each vertex is predicted based on the CNN backbone features. While the Deep Snake algorithm improves instance segmentation performance, the manually designed octagonal initial contour does not adequately enclose the instance and fails to fully utilize the information from the initial object detection and localization. The complex and variable paths of feature points during contour iterations lead to reduced instance segmentation performance. Furthermore, manually designing the initial contour is time-consuming, which reduces model inference speed. Furthermore, while existing Deep Snake algorithms can achieve real-time inference speeds on high-performance GPUs, in practical applications such as autonomous driving and security monitoring, they are often limited to devices with limited performance, such as embedded GPUs. In these cases, the Deep Snake algorithm can only achieve 5 FPS, far from meeting real-time requirements. Therefore, there is an urgent need to propose a contour-based image instance segmentation method to overcome the above-mentioned shortcomings of existing technologies. Summary of the Invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the technical defects existing in the existing technology, and propose a contour-based image instance segmentation method and system, which abandons the traditional method of manually designing the initial contour, and is based on network learning of the initial contour that is closer to the true value. While maintaining the algorithm segmentation accuracy, it has a smaller number of parameters and calculations, and has a faster inference speed, and can be applied to instance segmentation tasks in real scenarios using edge platforms.

[0006] To solve the above technical problems, the present invention provides a contour-based image instance segmentation method, comprising:

[0007] Input the target image into a lightweight feature extraction network to obtain the downsampled feature map, center point, and offset of the contour vertex relative to the center point of the target instance in the image;

[0008] Calculating an initial contour of the target instance based on the center point of the target instance and the offset of the contour vertex relative to the center point;

[0009] Performing global feature fusion on the downsampled feature map and the initial contour of the target instance to obtain a rough contour of the target instance;

[0010] A first contour deformation is performed on the rough contour of the target instance to obtain a preliminary instance contour, and a second contour deformation is performed on the preliminary instance contour to obtain a final instance contour.

[0011] In one embodiment of the present invention, the lightweight feature extraction network includes a feature extraction backbone network, a feature fusion module and a detection head module. The input target image passes through the feature extraction backbone network and the feature fusion module, and outputs a downsampled feature map of the target instance. The detection head module is used to regress the center point of the target instance and the offset of the contour vertex relative to the center point.

[0012] In one embodiment of the present invention, the feature extraction backbone network is ShuffleNetV2, and the 3×3 depth-wise separable convolution in the ShuffleNetV2 is replaced by a 5×5 depth-wise separable convolution.

[0013] In one embodiment of the present invention, a method for calculating an initial contour of a target instance based on a center point of the target instance and an offset of contour vertices relative to the center point includes:

[0014] The offset of each initial contour vertex relative to the center point is obtained as an ordered point set regression, expressed as:

[0015]

[0016] in, Indicates the horizontal offset of the i-th initial contour vertex, represents the vertical offset of the i-th initial contour vertex, and N represents the number of vertices of the initial contour;

[0017] The initial contour vertex coordinates are obtained by adding the center point coordinates and the offset, expressed as:

[0018]

[0019] in, represents the horizontal coordinate of the i-th initial contour vertex, Represents the vertical coordinate of the i-th initial contour vertex.

[0020] In one embodiment of the present invention, a method for performing global feature fusion on the downsampled feature map and the initial contour of the target instance includes:

[0021] Connect the vertex features of the initial contour into a vector of length N×C, where N is the number of vertices of the initial contour and C is the number of channels of the vertex features;

[0022] The offset of the contour vertex is predicted based on the vector. The offset of the contour vertex is a vector of length N×2, expressed as:

[0023]

[0024] in, Indicates the horizontal offset of the i-th rough contour vertex, represents the vertical offset of the ith rough contour vertex, and N represents the number of vertices of the rough contour;

[0025] The rough outline of the target instance is obtained by adding the offset and the initial contour coordinates, which is expressed as:

[0026]

[0027] in, represents the horizontal coordinate of the ith rough contour vertex, Represents the ordinate of the i-th rough contour vertex.

[0028] In one embodiment of the present invention, the method of performing the first contour deformation and the second contour deformation on the rough contour of the target instance is the same.

[0029] In one embodiment of the present invention, a method for performing a first contour deformation on a rough contour of the target instance includes:

[0030] Performing multi-scale feature extraction on the rough contour of the target instance to obtain contour vertex features at each scale of each layer;

[0031] Connect the contour vertex features at each scale of each layer in sequence to fuse the multi-scale contour features;

[0032] Process the fused contour features to generate the offset of the contour vertices;

[0033] The preliminary instance contour is obtained based on the offset and the vertices of the rough contour.

[0034] In addition, the present invention also provides a contour-based image instance segmentation system, comprising:

[0035] A feature extraction module is used to input the target image into a lightweight feature extraction network to obtain the downsampled feature map, center point, and offset of the contour vertex relative to the center point of the target instance in the image;

[0036] An initial contour generation module, configured to calculate an initial contour of a target instance based on a center point of the target instance and an offset of contour vertices relative to the center point;

[0037] A global feature fusion module is used to perform global feature fusion on the downsampled feature map and the initial contour of the target instance to obtain a rough contour of the target instance;

[0038] The contour deformation module is used to perform a first contour deformation on the rough contour of the target instance to obtain a preliminary instance contour, and perform a second contour deformation on the preliminary instance contour to obtain a final instance contour.

[0039] Furthermore, the present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-described method when executing the program.

[0040] Furthermore, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned method when executed by a processor.

[0041] The above technical solution of the present invention has the following advantages over the prior art:

[0042] The contour-based image instance segmentation method and system described in the present invention abandon the traditional method of manually designing initial contours, and instead learn initial contours that are closer to the true value based on network learning. While maintaining the algorithm segmentation accuracy, it has a smaller number of parameters and calculations, and has a faster inference speed. It can be applied to instance segmentation tasks in real scenarios using edge platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein

[0044] Figure 1 It is a flowchart of a contour-based image instance segmentation method proposed in the present invention.

[0045] Figure 2 It is a block diagram of a contour-based image instance segmentation method proposed in the present invention.

[0046] Figure 3 It is a structural diagram of the feature extraction backbone network proposed in this invention.

[0047] Figure 4 This is a schematic diagram of the convolution kernel expansion operation on the ShuffleNetV2 pre-trained model of 3×3 convolution proposed in the present invention.

[0048] Figure 5 It is a block diagram of the feature fusion module proposed in this invention.

[0049] Figure 6 It is a block diagram of the detection head module proposed in the present invention.

[0050] Figure 7 It is a schematic diagram of the specific implementation process of contour deformation of a rough contour proposed by the present invention.

[0051] Figure 8 It is a block diagram of the contour deformation module proposed in the present invention.

[0052] Figure 9 This is a comparison chart of the results of the method proposed in the present invention and the existing method.

[0053] Figure 10 It is the result diagram of the method proposed in the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0055] Reference Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a contour-based image instance segmentation method, comprising the following steps:

[0056] Step S1: Input the target image into the lightweight feature extraction network to obtain the downsampled feature map, center point and offset of the contour vertex relative to the center point of the target instance in the image;

[0057] Step S2: calculating an initial contour of the target instance based on the center point of the target instance and the offset of the contour vertex relative to the center point;

[0058] Step S3: performing global feature fusion on the downsampled feature map and the initial contour of the target instance to obtain a rough contour of the target instance;

[0059] Step S4: performing a first contour deformation on the rough contour of the target instance to obtain a preliminary instance contour, and performing a second contour deformation on the preliminary instance contour to obtain a final instance contour.

[0060] The contour-based image instance segmentation method described in the present invention abandons the traditional method of manually designing initial contours, and instead learns initial contours that are closer to the true value based on network learning. While maintaining the algorithm segmentation accuracy, it has a smaller number of parameters and computational complexity, and has a faster inference speed. It can be applied to instance segmentation tasks in real scenarios using edge platforms.

[0061] Among them, in step S1, the lightweight feature extraction network includes a feature extraction backbone network, a feature fusion module and a detection head module. The input target image passes through the feature extraction backbone network and the feature fusion module, and outputs a downsampled feature map of the target instance. The detection head module is used to regress and obtain the center point of the target instance and the offset of the contour vertex relative to the center point.

[0062] As an example, the feature extraction backbone network can adopt ShuffleNetV2. ShuffleNetV2 adopts lightweight group convolution and channel rearrangement technology, which reduces the amount of calculation and parameters, has lower computational complexity, and can run on edge devices. At the same time, the channel rearrangement technology separates calculation and communication, thereby reducing the memory bandwidth bottleneck and further improving the inference speed. While maintaining a low computational complexity, ShuffleNetV2 can provide model accuracy comparable to or even better than the traditional network structure. Therefore, this embodiment preferably adopts ShuffleNetV2 as the backbone feature extraction network, and the backbone network structure is as follows: Figure 3 As shown in the left half, the lightweight network ShuffleNetV2 as the backbone feature extraction network will affect the depth and receptive field of the model, and the Prediction Head mainly relies on the heat map to regress the target center point, so the receptive field of the model is very important. The size of the receptive field is affected by the convolution kernel. The larger the convolution kernel size, the larger the receptive field. However, as the size of the convolution kernel increases, the number of network parameters will increase greatly. The VGG network uses multiple small-scale convolution kernels to stack into large-scale convolutions. For example, two 3x3 convolution kernels are used to obtain the receptive field of a 5x5 convolution kernel. Under the same receptive field, the number of parameters and computational complexity of two 3x3 convolutions are only 18 / 25 of that of one 5x5. That is, this embodiment replaces the 3×3 depth-separable convolution in ShuffleNetV2 with a 5×5 depth-separable convolution, as shown in Figure 3 As shown in the K5-Stage2, K5-Stage3, and K5-Stag / 3 modules, the receptive field is doubled.

[0063] The existing ShuffleNetV2 pre-trained model based on ImageNet uses 3×3 depth convolution, and there is a lack of pre-trained models based on 5×5 depth convolution. In order to make the model converge faster, Figure 4 As shown, this embodiment performs a convolution kernel expansion operation on the 3×3 convolution ShuffleNetV2 pre-trained model to obtain a 5×5 large convolution kernel ShuffleNetV2 pre-trained model.

[0064] As an example, ShuffleNetV2-k5 is used as the feature extraction backbone network to downsample the input image by 2, 4, 8, and 16 times, respectively, to obtain the corresponding feature maps. In order to fuse the features of different downsampling multiples, this embodiment performs iterative deep feature aggregation on the feature maps of different stages, and finally obtains an output that is 4 times the input image, which is used for the detection head to regress the target center point coordinates and width and height. Figure 5 As shown in the figure, each round of deep feature aggregation calculates the lower right right triangle from small to large, and fuses them to obtain 16x, 8x, and 4x downsampled feature maps, respectively. The 4x downsampled feature map is used as the output. At the same time, some convolutions in the deep feature aggregation module are replaced with deformable convolutions to improve the model's perception of small objects.

[0065] As an example, in order to obtain the target instance center coordinates and initial contour offset, the following is used: Figure 6 The prediction head module shown in Figure 1 obtains the instance center coordinates and initial contour offsets through two full convolutional network regressions for subsequent initial contour calculations. Figure 6 As shown in Figure 1, the downsampled feature map is input into the prediction head modules CT-Head and Offset-Head to obtain the instance center coordinates and contour offset (i.e., the offset of the contour vertex relative to the center point), respectively.

[0066] Among them, in step S2, the initial contour method learned by the network changes the decoding method of the target detection results. The generation of the BoundingBox requires complex calculations, while the initial contour learned by the network no longer requires complex deformation calculations of the BoundingBox to obtain the initial contour. The initial contour is directly obtained by the center point of the network learning instance and the offset of the contour vertex relative to the center point. The features of each vertex of the initial contour are then fused to deform the initial contour to obtain a rough contour closer to the groundtruth for the subsequent Snake contour deformation.

[0067] Furthermore, the initial contour method learned by the network does not regress the width and height of the target detection box in addition to regressing the coordinates of the target center point when outputting the target detection method. Based on the center point feature, the offset of each initial contour vertex relative to the center point is directly regressed as an ordered point set, expressed as:

[0068]

[0069] in, Indicates the horizontal offset of the i-th initial contour vertex, represents the vertical offset of the i-th initial contour vertex, N is the number of vertices of the initial contour, and N is set to 128 in the present invention. The coordinates of the initial contour vertex are obtained by adding the center point coordinates and the offset, and are expressed as:

[0070]

[0071] in, represents the horizontal coordinate of the i-th initial contour vertex, Represents the vertical coordinate of the i-th initial contour vertex.

[0072] Therefore, the initial contour learned by the network is closer to the true contour. In addition, the deformation path of the initial contour learned by the network is from the center point to the vertex of the initial contour, and the vertex deformation path diverges from the inside to the outside. There are no unfavorable intersections between the deformation paths that affect the convergence of the initial contour.

[0073] Among them, in step S3, there is still a large gap between the initial contour obtained by only regressing the instance center point features and the final contour. The initial contour cannot be directly deformed to obtain the instance contour. It is also difficult to fully characterize the contour features based on the local features of a single contour vertex or several adjacent vertices, and the contour cannot be effectively deformed. DeepSnake uses the Snake contour deformation module to aggregate local features, fuse the features of several adjacent vertices, obtain relatively complete contour features, and to a certain extent complete the global contour information. However, if the Snake module is used multiple times for contour deformation, although the contour features can be better aggregated, it increases the complexity of the model and reduces the speed of model reasoning. At the same time, the large errors formed during the contour deformation process are not well corrected.

[0074] Therefore, this embodiment adopts the global contour feature aggregation mechanism to simply and efficiently improve the network's ability to learn contour global features and deform the rough contour based on all contour vertex features. The specific implementation process is as follows Figure 7 As shown in Figure 1, N rough contour vertex features are first concatenated into a vector of length N×C (where C is the number of channels of the vertex feature). The vector is then input into the global contour deformation module to predict the contour vertex offset, where the number of channels in the hidden layer and output layer of the global contour deformation module is N×2, which is Figure 7 As shown in the circle in the middle. The offset of the contour vertex is a vector of length N×2, expressed as:

[0075]

[0076] in, Indicates the horizontal offset of the i-th rough contour vertex, represents the vertical offset of the i-th rough contour vertex, and N is the number of contour vertices. Finally, the offset and the rough contour coordinates are added to obtain the adjusted rough contour of the target instance, which is expressed as:

[0077]

[0078] in, represents the horizontal coordinate of the ith rough contour vertex, Represents the ordinate of the i-th rough contour vertex, and N is the number of contour vertices.

[0079] In step S4, the feature points of the instance contour are different from ordinary image pixel features. Instead, they are similar to a discrete signal. Compared with traditional two-dimensional convolution, circular convolution is more suitable for calculating such discrete features. Therefore, Deep Snake uses 1D circular convolution to calculate the contour vertices. Deep Snake regards the vertices on the contour as a set of discrete signals (f N ) i =f i Mod N, use one-dimensional circular convolution to calculate the contour vertex features:

[0080]

[0081] Among them, f N is the contour vertex signal, k is the 1D convolution kernel, * is the 1D convolution operation, i represents the sequence number of the contour vertex, r represents the size of the circular convolution kernel, and in the present invention, r is determined to be 9, j∈[-r,r].

[0082] Based on one-dimensional circular convolution, Deep Snake designs contour deformation modules such as Figure 8 As shown in the figure, it consists of three modules: the backbone network, the fusion module, and the prediction head. First, the rough contour is input into the backbone network. The backbone network has 8 recurrent convolution layers for multi-scale feature information extraction, and the features of each vertex are output to the fusion module. The fusion module connects the contour vertex features at each scale of each layer in sequence, and then forwards them using a one-dimensional convolution kernel. The feature information obtained is then compressed through maximum pooling to fuse the multi-scale contour features. Finally, in the prediction head module, three convolution layers are used to process the fused features, generate offsets and calculate the rough contour vertices to obtain the contour after a round of contour deformation. The contour after one round of contour deformation is then subjected to a second round of contour deformation according to the same steps.

[0083] In summary, according to the overall model structure design, the entire network learning process mainly consists of two parts: target center learning and instance contour deformation learning. Therefore, the design of the loss function mainly includes two parts, one for the initial target center prediction and the other for the instance contour deformation.

[0084] In the experiment of this invention, the confidence loss function used for target center point prediction refers to CenterNet, and a Heatmap is generated to predict the target center point information. The center point is represented as:

[0085]

[0086] Where C is the number of categories, R represents the downsampling step size, W and H represent the width and height of the original image, and each category has a corresponding heatmap. When a certain coordinate in the image contains the target center point, a key point will be generated at that coordinate, represented by a Gaussian circle. For each category C, there is a target center point position truth value p∈R 2 , calculate the approximate value of p through training Apply a Gaussian transform:

[0087]

[0088] Map all true values to the heatmap, where X and Y represent the horizontal and vertical coordinates of the predicted target center point. represents the approximate value of p in the horizontal and vertical directions, Indicates the standard deviation, thus obtaining the approximate value of the center position. FocalLoss is used as the loss function during training, expressed as Loss ct .

[0089]

[0090] Among them, α and β represent hyperparameters. In experiments, α is generally set to 2 and β is set to 4 to balance positive and negative samples and difficult and easy samples; Y xyc Indicates the distance weight of (x, y) from the true value of the target center point. The closer its value is to 1, the closer (x, y) is to the actual center point of the target. Indicates the probability that (x, y) is predicted to be the center point of the target; N indicates the number of key points.

[0091] The loss function used for contour deformation uses SmoothL1, which consists of three parts: initial contour loss, rough contour loss, and iterative deformation contour loss, which are defined as:

[0092]

[0093]

[0094]

[0095] Where N is the number of contour vertices, is the true value of the contour label, is the predicted initial contour vertex, are the predicted rough contour vertices, is the predicted iterative deformed contour vertex, which includes two iterative optimizations. Therefore, the overall loss function is:

[0096] L all =L ct +αL init +βL coarse +L iter1 +L iter2 (3-11)

[0097] Among them, α and β are the weights of the initial contour and rough contour deformation loss values, L all Represents the overall loss function, L ct Represents the loss function of center point prediction, L init Represents the loss function for generating the initial contour, L coarse Represents the loss function for generating rough contours, L iter1 、L iter2 represents the loss function for the first and second contour optimization.

[0098] In the experiments of the present invention, the Deep Snake algorithm was used as a reference for the setting of the hyperparameters α and β. However, in order to make the hyperparameters α and β values more suitable for the algorithm designed by the present invention, α and β were set to 0.05, 0.1, and 0.2 when training the SBDval dataset. When α and β were set to 0.1, the average accuracy was higher than the other two settings in the second half of the training. At the end of the training, the highest average accuracy was achieved when α and β were set to 0.1. Therefore, in the subsequent experiments of the present invention, α and β were both set to 0.1. Therefore, during the model training process, α and β were both set to 0.1, unless otherwise stated.

[0099] To improve the accuracy and speed of the instance segmentation algorithm, this chapter makes improvements to the backbone network and the contour initialization method. To verify the effectiveness of these improvements, ablation experiments were conducted on the SBD dataset. The model was trained for 150 rounds, with the learning rate halved at the 80th and 120th epochs. The ablation experiment results are shown in Table 1. After using the network-learned contour initialization method, the instance segmentation accuracy increased by 1.8 AP. After using the traditional ShuffleNetV2 backbone network, the model inference speed increased by 8 FPS compared to DeepSnake, but the segmentation accuracy decreased significantly. Finally, by replacing the convolution kernel in ShuffleNetV2 with a 5*5 kernel, the model achieved a larger receptive field, improved the segmentation accuracy by 0.5 AP compared to DeepSnake, reduced the number of parameters by approximately 87.2%, reduced the computational effort by approximately 78.3%, and increased the inference speed by 7.5 FPS, achieving a good balance between accuracy and speed.

[0100] Table 1 Ablation experiments on the SBD dataset

[0101]

[0102] As shown in Table 2, all four algorithms performed instance segmentation on the PASCAL SBD dataset on 512x512 pixel images. Compared to Deep Snake, the proposed method improves segmentation accuracy by 7.5 fps. This makes it highly competitive with other algorithms in terms of accuracy and speed.

[0103] Table 2 Comparison with other methods on SBD val dataset

[0104]

[0105] like Figure 9 As shown, the fourth row shows that the algorithm proposed in the present invention can accurately identify the number and category of instances in the image and segment the contours. The FCIS algorithm can roughly give the contours of the instances, but there are some false detections, such as the fourth picture in the first row. The ESE-20 algorithm can identify different instances and their categories, but the segmented contours are not close enough to the actual contours. Especially in the fifth picture, probably because the train occupies most of the area, ESE-20 cannot give the instance contours. The segmented contours shown in the fourth row greatly reduce false detections and missed detections. For example, among the three girls in the third column, the Deep Snake algorithm has false detections, but the algorithm proposed in the present invention can accurately identify the three girls and clearly segment the contours.

[0106] like Figure 10 As shown, (a) is the original image, (b) is the rough contour of the instance calculated by the initial contour regression and global contour feature fusion module of network learning, and (c) is the final contour predicted by the Snake contour deformation module twice. The visualization results show that the rough contour is very close to the true contour of the instance, and is also very close to the instance contour obtained by the final iteration, indicating that the initial contour regression and global contour feature fusion module of network learning proposed in the present invention has good feature learning and contour deformation capabilities, provides more accurate input for the Snake contour deformation module, and enables the model to obtain more accurate instance contours through fewer iterative calculations.

[0107] A contour-based image instance segmentation system disclosed in an embodiment of the present invention is introduced below. The contour-based image instance segmentation system described below and the contour-based image instance segmentation method described above can correspond to each other.

[0108] The present invention also provides a contour-based image instance segmentation system, comprising:

[0109] A feature extraction module is used to input the target image into a lightweight feature extraction network to obtain the downsampled feature map, center point, and offset of the contour vertex relative to the center point of the target instance in the image;

[0110] An initial contour generation module, configured to calculate an initial contour of a target instance based on a center point of the target instance and an offset of contour vertices relative to the center point;

[0111] A global feature fusion module is used to perform global feature fusion on the downsampled feature map and the initial contour of the target instance to obtain a rough contour of the target instance;

[0112] The contour deformation module is used to perform a first contour deformation on the rough contour of the target instance to obtain a preliminary instance contour, and perform a second contour deformation on the preliminary instance contour to obtain a final instance contour.

[0113] The contour-based image instance segmentation system of this embodiment is used to implement part of the embodiment of the aforementioned contour-based image instance segmentation method. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part and will not be introduced in detail here.

[0114] In addition, since the contour-based image instance segmentation system of this embodiment is used to implement the aforementioned contour-based image instance segmentation method, its function corresponds to that of the aforementioned method and will not be described in detail here.

[0115] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0119] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A contour-based image instance segmentation method, characterized by: include: Input the target image into a lightweight feature extraction network to obtain the downsampled feature map, center point, and offset of the contour vertex relative to the center point of the target instance in the image; The initial contour of the target instance is calculated based on the center point of the target instance and the offset of the contour vertex relative to the center point. The offset of each initial contour vertex relative to the center point is obtained by regression as an ordered point set, which is expressed as , Indicates the The horizontal offset of the initial contour vertex, Indicates the The vertical offset of the initial contour vertex, Indicates the number of vertices of the initial contour; the initial contour vertex coordinates are obtained by adding the center point coordinates and the offset, expressed as , Indicates the The horizontal coordinates of the initial contour vertices, Indicates the The ordinate of the initial contour vertex; Perform global feature fusion on the downsampled feature map and the initial contour of the target instance to obtain a rough contour of the target instance; including connecting the vertex features of the initial contour into a length of vector, is the number of channels of the vertex feature; the offset of the contour vertex is predicted based on the vector, and the offset of the contour vertex is of length The vector is represented as , Indicates the The horizontal offset of the rough contour vertex, Indicates the The vertical offset of the rough contour vertex is added to the initial contour coordinates to obtain the rough contour of the target instance, which is expressed as , Indicates the The horizontal coordinates of the vertices of the rough contour, Indicates the The vertical coordinates of the vertices of the rough contour; A first contour deformation is performed on the rough contour of the target instance to obtain a preliminary instance contour, and a second contour deformation is performed on the preliminary instance contour to obtain a final instance contour.

2. The contour-based image instance segmentation method according to claim 1, wherein: The lightweight feature extraction network includes a feature extraction backbone network, a feature fusion module and a detection head module. The input target image passes through the feature extraction backbone network and the feature fusion module, outputs a downsampled feature map of the target instance, and uses the detection head module to regress the center point of the target instance and the offset of the contour vertex relative to the center point.

3. The contour-based image instance segmentation method according to claim 2, wherein: The feature extraction backbone network is ShuffleNetV2, and the 3×3 depth-separable convolution in ShuffleNetV2 is replaced by a 5×5 depth-separable convolution.

4. The contour-based image instance segmentation method according to claim 1, wherein: The method of performing the first contour deformation and the second contour deformation on the rough contour of the target instance is the same.

5. The contour-based image instance segmentation method according to claim 1 or 4, characterized in that: The method for performing a first contour deformation on the rough contour of the target instance includes: Performing multi-scale feature extraction on the rough contour of the target instance to obtain contour vertex features at each scale of each layer; Connect the contour vertex features at each scale of each layer in sequence to fuse the multi-scale contour features; Process the fused contour features to generate the offset of the contour vertices; The preliminary instance contour is obtained based on the offset and the vertices of the rough contour.

6. A contour-based image instance segmentation system, characterized by: include: A feature extraction module is used to input the target image into a lightweight feature extraction network to obtain the downsampled feature map, center point, and offset of the contour vertex relative to the center point of the target instance in the image; The initial contour generation module is used to calculate the initial contour of the target instance based on the center point of the target instance and the offset of the contour vertex relative to the center point; the offset of each initial contour vertex relative to the center point is obtained by regression as an ordered point set, expressed as , Indicates the The horizontal offset of the initial contour vertex, Indicates the The vertical offset of the initial contour vertex, Indicates the number of vertices of the initial contour; the initial contour vertex coordinates are obtained by adding the center point coordinates and the offset, expressed as , Indicates the The horizontal coordinates of the initial contour vertices, Indicates the The ordinate of the initial contour vertex; The global feature fusion module is used to perform global feature fusion on the downsampled feature map and the initial contour of the target instance to obtain a rough contour of the target instance; including connecting the vertex features of the initial contour into a length of vector, is the number of channels of the vertex feature; the offset of the contour vertex is predicted based on the vector, and the offset of the contour vertex is of length The vector is represented as , Indicates the The horizontal offset of the rough contour vertex, Indicates the The vertical offset of the rough contour vertex is added to the initial contour coordinates to obtain the rough contour of the target instance, which is expressed as , Indicates the The horizontal coordinates of the vertices of the rough contour, Indicates the The vertical coordinates of the vertices of the rough contour; The contour deformation module is used to perform a first contour deformation on the rough contour of the target instance to obtain a preliminary instance contour, and perform a second contour deformation on the preliminary instance contour to obtain a final instance contour.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images