Training method, segmentation method and device of multi-scale object instance segmentation model

Abstract

A training method, segmentation method, and apparatus for a multi-scale object instance segmentation model are disclosed. A feature extraction network extracts a first fused feature map from the input image. This first fused feature map is input into a target box prediction network to obtain predicted target boxes and corresponding attention maps in the input image. The predicted target boxes are then filtered to obtain predicted instance target boxes. The first fused feature map is input into the segmentation module of a masking network to obtain a predicted score map. The attention map, predicted score map, and predicted instance target boxes are input into the fusion module of the masking network to output the object instance segmentation result. This invention focuses on image feature extraction and target box prediction. It is simple to use, automates the process, and combines high-level and low-level feature information of object instances through a masking network. This enables the effective prediction of dense, pixel-by-pixel position-sensitive object instance features with fewer parameters, significantly improving segmentation quality and achieving real-time fast inference.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a training method, segmentation method, and apparatus for a multi-scale object instance segmentation model. Background Technology

[0002] Object segmentation is a machine vision segmentation method for detecting all target objects in the imaging of industrial products. It is widely used in robot vision localization, product quality parameter analysis (including shape, quantity, area, center point, etc.), robot grasping and other fields.

[0003] In industrial production, object instance segmentation is challenging for applications with diverse product categories and varying sizes. Tasks such as product recognition and positioning, quality analysis, grasping, and sorting rely on accurate segmentation of target objects. Furthermore, various industrial scenarios have high demands for segmentation metrics. However, the significant differences in size, shape, and scale between objects add extra complexity to object instance segmentation, easily leading to missed or false detections of extremely large or excessively large-scale objects. Manufacturing enterprises have high requirements for the segmentation accuracy of object instances of different scales for subsequent product quality analysis, precise positioning, and other tasks. Summary of the Invention

[0004] The main technical problem this invention addresses is how to achieve accurate segmentation of multi-scale object instances.

[0005] According to a first aspect, one embodiment provides a training method for a multi-scale object instance segmentation model, the multi-scale object instance segmentation model including a feature extraction network, a bounding box prediction network, and a masking network, wherein the masking network includes a segmentation module and a fusion module, and the training method includes:

[0006] The method involves acquiring an input image containing one or more object instances and annotation data, wherein the annotation data includes the location information and category information of the marked target boxes in the input image and the corresponding binary image of the input image.

[0007] The input image is input into the feature extraction network to obtain a first fusion feature map with multiple layers having the same channel but different resolutions;

[0008] The first fusion feature maps with the same channels but different resolutions are input into the target box prediction network to obtain the position information, category information, and centrality of the predicted target boxes in each layer of the first fusion feature map, as well as the attention map corresponding to each layer of the first fusion feature map. The attention map of each layer has the same resolution as the first fusion feature map of the corresponding layer. The centrality is used to characterize the centrality of each pixel in the first fusion feature map in the corresponding predicted target box.

[0009] Based on the predicted target box's location information, category information, and centrality, as well as the labeled target box's location information and category information in the labeled data, a target box prediction network loss function is constructed.

[0010] The first fusion feature map of the multilayer is input into the segmentation module of the mask network to obtain a predicted segmentation map and a predicted score map. The predicted segmentation map has 2 channels and has the same resolution as the first layer first fusion feature map of the first fusion feature map of the multilayer. The predicted score map has 4 channels and has the same resolution as the first layer first fusion feature map of the first fusion feature map of the multilayer.

[0011] The attention maps, prediction score maps, and marked target boxes of the multi-layered network are input into the fusion module of the masking network to obtain the instance prediction image;

[0012] Based on the predicted segmentation map and the labeled binary map corresponding to the scale-transformed input image, a segmentation module loss function is constructed; wherein, the labeled binary map corresponding to the scale-transformed input image has the same resolution as the predicted segmentation map;

[0013] Based on the instance prediction image and the binarized image corresponding to the marked target box in the labeled binary image of the scale-transformed input image, a fusion module loss function is constructed; wherein, the binarized image and the instance prediction image have the same resolution;

[0014] Based on the target bounding box prediction network loss function, the segmentation module loss function, and the fusion module loss function, a total loss function is constructed, and the multi-scale object instance segmentation model is trained based on the total loss function to obtain the corresponding model parameters.

[0015] According to the second aspect, one embodiment provides a method for instance segmentation of multi-scale objects, comprising:

[0016] Obtain an input image that includes one or more object instances;

[0017] The input image is input into a feature extraction network to obtain a first fusion feature map with multiple layers having the same channel but different resolutions.

[0018] The first fusion feature maps with the same channels but different resolutions are input into the target box prediction network to obtain the position information, category information and centrality of the predicted target boxes in the first fusion feature maps of each layer, as well as the attention map corresponding to the first fusion feature map of each layer, wherein the attention map of each layer has the same resolution as the first fusion feature map of the corresponding layer.

[0019] For each pixel in the first fusion feature map of each layer, all predicted target boxes are filtered to determine whether the predicted target box for each pixel in the first fusion feature map of each layer is a predicted instance target box; if it is a predicted instance target box, the corresponding pixel is taken as the predicted pixel.

[0020] The first fused feature map from multiple layers is input into the segmentation module of the mask network to obtain the predicted score map;

[0021] The attention map, the prediction score map, and the predicted instance target boxes are input into the fusion module of the masking network to obtain an instance prediction image. The instance prediction image contains the pixel segmentation results of the input image with a preset resolution and the same number of predicted instance target boxes.

[0022] The pixel segmentation results of all predicted instance target boxes with preset resolutions are scaled to the size of the corresponding predicted instance target boxes to obtain the pixel segmentation results of all predicted instance target boxes in the input image. Pixels in the input image that do not fall within the predicted instance target boxes are defaulted to background pixels to obtain the final object instance segmentation result.

[0023] According to a third aspect, one embodiment provides an instance segmentation apparatus for multi-scale objects, comprising:

[0024] Image acquisition device, used to acquire input images;

[0025] A processor for executing the instance segmentation method for multi-scale objects as described in any of the above embodiments.

[0026] According to a fourth aspect, one embodiment provides a computer-readable storage medium, characterized in that the medium stores a program that can be executed by a processor to implement the method as described in any of the above embodiments.

[0027] According to the training method, segmentation method, and apparatus of the multi-scale object instance segmentation model in the above embodiments, the multi-scale object instance segmentation model includes a feature extraction network, a target box prediction network, and a masking network. First, the feature extraction network effectively extracts multi-layer first fusion feature maps of the input image. Then, the multi-layer first fusion feature maps are input into the target box prediction network to obtain predicted target boxes in the input image and corresponding attention maps for each layer of the first fusion feature map. All predicted target boxes predicted for each pixel in each layer of the first fusion feature map are filtered to obtain predicted instance target boxes. Then, the first fusion feature maps are input into the segmentation module of the masking network to obtain a prediction score map. Finally, the attention map, prediction score map, and predicted instance target boxes are input into the fusion module of the masking network. The block predicts multi-scale object instance prediction images and scales the pixel segmentation results of the predicted instance target boxes in the instance prediction images to obtain the pixel segmentation results of all predicted instance target boxes in the input image. Pixels in the input image that do not fall within the predicted instance target boxes are defaulted to background pixels, resulting in the final object instance segmentation result. Thus, this invention does not require setting anchor box parameters and directly extracts predicted target boxes based on image features. It is simple to use, easy to operate, and runs automatically. Furthermore, by combining high-level and low-level feature information of object instances through a masking network, it can effectively predict dense, pixel-by-pixel position-sensitive object instance features with fewer parameters, greatly improving the object instance segmentation quality and achieving real-time fast inference. Attached Figure Description

[0028] Figure 1 A schematic diagram of the structure of a multi-scale instance segmentation model according to one embodiment;

[0029] Figure 2 A flowchart illustrating a training method for a multi-scale object instance segmentation model according to one embodiment;

[0030] Figure 3 This is a flowchart of a method for extracting a first fused feature map using a feature extraction network according to one embodiment;

[0031] Figure 4 This is a schematic diagram illustrating the structural flow of a multi-scale aggregation network layer according to one embodiment.

[0032] Figure 5 This is a schematic diagram illustrating the structural flow of a target bounding box prediction network according to one embodiment.

[0033] Figure 6 A flowchart illustrating a method for a bounding box prediction network to predict bounding boxes and corresponding layer attention maps, according to one embodiment;

[0034] Figure 7This is a schematic diagram showing the label values ​​calculated by mapping pixels in the first fused feature map to their positions in the input image and their corresponding bounding boxes.

[0035] Figure 8 This is a schematic diagram of the structural flow of a segmentation module according to one embodiment;

[0036] Figure 9 A flowchart illustrating a method for outputting instance prediction images by a fusion module in one embodiment;

[0037] Figure 10 A flowchart illustrating an instance segmentation method for multi-scale objects according to one embodiment;

[0038] Figure 11 A schematic diagram illustrating the effect of multi-scale object instance segmentation;

[0039] Figure 12 An instance segmentation device for multi-scale objects is provided as one embodiment. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0041] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0042] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. Unless otherwise specified, the terms "connection" and "linkage" used in this application include both direct and indirect connections (linkages).

[0043] Currently, for multi-scale object instance segmentation, industrial applications typically employ a pre-defined anchor box approach. This involves first acquiring information such as the size and distribution of target objects in the product data. Then, based on this pre-acquired information, parameters such as the anchor box's scale and aspect ratio are set to cover as many target object scale distributions as possible. Finally, the instance segmentation model is trained and inferred. For industrial scenarios with few product types and small scale differences, or large scale differences but relatively fixed sizes, this pre-defined anchor box approach can achieve multi-scale object instance segmentation. However, for industrial scenarios with many product types, large scale differences, and inconsistent sizes, this approach requires adding many prior parameters such as anchor box scale and aspect ratio to accommodate the data distribution characteristics. This increases the amount of unnecessary computation, making the instance segmentation model difficult to train. Furthermore, considering the computational performance limitations of the application, real-time requirements cannot be met. Therefore, while prioritizing inference speed, the segmentation quality of object instances decreases, making it prone to missed detections and false detections.

[0044] To address the aforementioned issues, this invention provides a training method and a segmentation method for a multi-scale object instance segmentation model. This method effectively combines low-level fine-grained semantic information with high-level instance features to achieve accurate segmentation of multi-scale object instances in an end-to-end manner, thereby improving segmentation performance.

[0045] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of a multi-scale instance segmentation model according to one embodiment. The multi-scale instance segmentation model provided in this embodiment includes a feature extraction network G, a bounding box prediction network V, and a masking network P. The masking network P includes a segmentation module S and a fusion module B. The input image F is input into the feature extraction network G to obtain a multi-layer first fused feature map α. j Where j represents the j-th layer of the multi-layer first fusion feature map; multi-layer first fusion feature map α j The input is fed into the target box prediction network V to obtain the predicted target box β and the attention map a of the corresponding layer. j The first fusion feature map α of the multilayer j Predicted target box β and attention map of the corresponding layer a j The input masking network P predicts the output object instance segmentation result R, completing the object instance segmentation result R for multi-scale objects; wherein, in the object instance segmentation result R, the pixels in the target box are classified as background pixels and object instance pixels, and the pixels falling outside the target box are background pixels.

[0046] The training and inference methods of the multi-scale object instance segmentation model are explained below based on the structure of each network in the multi-scale instance segmentation model.

[0047] Please refer to Figure 2 , Figure 2 The flowchart below shows a training method for a multi-scale object instance segmentation model according to one embodiment, hereinafter referred to as the model training method. The model training method includes steps 101 to 109, which are described in detail below.

[0048] Step 101: Obtain an input image F containing one or more object instances and annotation data. The annotation data includes the location information and category information of the marked target boxes in the input image F, and the corresponding binary image of the input image F. The binary image is used to label all pixels in the input image F pixel by pixel to classify all pixels in the input image F into background pixels and object instance pixels. Different gray values ​​can be used to characterize whether a pixel is a background pixel or an object instance pixel. For example, if a pixel in the input image F is labeled as 0, it is a background pixel; if a pixel is labeled as 1, it is an object instance pixel. It should also be noted that the category of an object instance is determined by the category information of its target box. For pixels within the target box, it is only necessary to distinguish whether they are background pixels or object instance pixels. In this embodiment, the annotation data can be obtained by manually labeling the real target boxes in the input image F and the object instance pixels and background pixels within the target boxes, or it can be obtained directly from a sample database.

[0049] Step 102: Input the input image F into the feature extraction network G to obtain a first fusion feature map with multiple layers having the same channel but different resolutions.

[0050] Step 103: Input the first fusion feature maps with multiple layers having the same channel but different resolutions into the target box prediction network V to obtain the position information, category information and centrality of the predicted target boxes in each layer of the first fusion feature map, as well as the attention map corresponding to each layer of the first fusion feature map, wherein each layer of attention map has the same resolution as the first fusion feature map of the corresponding layer; wherein, the centrality is used to characterize the centrality of each pixel in the first fusion feature map in the corresponding predicted target box.

[0051] Step 104: Based on the predicted target box location information, category information, and centrality, as well as the labeled target box location information and category information in the labeled data, construct the target box prediction network loss function.

[0052] Step 105: Input the multi-layer first fusion feature map into the segmentation module S of the mask network P to obtain the predicted segmentation map and the predicted score map. The predicted segmentation map has 2 channels and has the same resolution as the first layer first fusion feature map in the multi-layer first fusion feature map. The predicted score map has 4 channels and has the same resolution as the first layer first fusion feature map in the multi-layer first fusion feature map.

[0053] Step 106: Input the multi-layer attention map, prediction score map, and labeled target bounding boxes into the fusion module B of the masking network P to obtain the instance prediction image. The labeled target bounding boxes input into the fusion module B of the masking network P contain both location and category information; for ease of explanation, this will not be repeated hereafter.

[0054] Step 107: Construct a segmentation module loss function based on the predicted segmentation map and the labeled binary map corresponding to the scaled input image F; wherein, the labeled binary map corresponding to the scaled input image F has the same resolution as the predicted segmentation map. In this embodiment, the scale transformation is used to scale the image data to a specified resolution. This image data can be a labeled binary map from the labeled data, or various feature maps.

[0055] Step 108: Construct the fusion module loss function based on the instance prediction image and the binarized image corresponding to the marked target box in the labeled binary image of the scaled input image F; wherein the binarized image and the instance prediction image have the same resolution.

[0056] Step 109: Construct a total loss function based on the target bounding box prediction network loss function, the segmentation module loss function, and the fusion module loss function, and train the multi-scale object instance segmentation model based on the total loss function to obtain the corresponding model parameters.

[0057] In one embodiment, the feature extraction network G includes a coarse extraction layer and a fine extraction layer. The coarse extraction layer includes two 3x3 standard convolutional layers, and the fine extraction layer includes multiple multi-scale aggregation network layers. Please refer to [reference needed]. Figure 3 , Figure 3 A flowchart of a method for extracting a first fused feature map for a feature extraction network G, comprising steps 1021 to 1023.

[0058] Step 1021: Input image F is fed into the coarse extraction layer and subjected to two 3x3 standard convolutions to obtain the initial feature map. In the coarse extraction layer, preliminary features of the input image F are extracted.

[0059] Step 1022: The initial feature map is input into the fine extraction layer and processed through multiple multi-scale aggregation network layers to obtain multiple high-level feature maps with different channels and resolutions. In the fine extraction layer, effective features at different scales of the input image F are extracted through the processing of multiple multi-scale aggregation network layers.

[0060] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure and flow of a multi-scale aggregation network layer. Assuming the input feature of the multi-scale aggregation network layer is `input`, for the first multi-scale aggregation network layer, the input feature `input` is the initial feature map output by the coarse extraction layer. For other multi-scale aggregation network layers besides the first layer, the input feature `input` is the high-level feature map of the corresponding layer output by the previous multi-scale aggregation network layer. Figure 4 For example, for a single multi-scale aggregation network layer, the multi-scale aggregation network layer is processed as follows:

[0061] (1) The input feature is processed by 1×1 convolution (C1x1) and channel splitting to obtain multiple split feature maps I1, I2, I3, and I4. Those skilled in the art will understand that the number of split feature maps is not limited to 4 and can be adjusted according to the actual application scenario and requirements.

[0062] (2) The first separate feature map among the multiple separate feature maps is taken as the first feature sub-map. The second separate feature map among the multiple separate feature maps is processed by a 3×3 standard convolution (C3x3) to obtain the second feature sub-map. The i-th separate feature map and the (i-1)-th feature sub-map among the multiple separate feature maps are then subjected to Hadamard addition and then processed by a 3×3 standard convolution (C3x3) to obtain the i-th feature sub-map, thus obtaining multiple feature sub-maps, where i is an integer greater than or equal to 3. For example, in... Figure 4 In the process, the separated feature map I1 is used directly as the first feature sub-map J1 without any processing. The separated feature map I2 is processed by a 3×3 standard convolution (C3x3) to obtain the second feature sub-map J2. The result of copying the separated feature map I2 is combined with the separated feature map I3 by performing Hadamard addition and then a 3×3 standard convolution (C3x3) to obtain the third feature sub-map J3. The result of copying the separated feature map I3 is combined with the separated feature map I4 by performing Hadamard addition and then a 3×3 standard convolution (C3x3) to obtain the fourth feature sub-map J4.

[0063] (3) After concatenating the channels of multiple feature sub-maps J1, J2, J3 and J4, perform 1×1 convolution (C1x1) to obtain the first feature map.

[0064] (4) Perform Hadamard addition on the input feature and the first feature map to obtain a higher-level feature map output.

[0065] As described above, for the first multi-scale aggregation network layer, the input feature is the initial feature map output by the coarse extraction layer. For other multi-scale aggregation network layers besides the first layer, the input feature is the high-level feature map of the corresponding layer output by the previous multi-scale aggregation network layer.

[0066] In this embodiment, the number of separated feature maps obtained by channel averaging of the input features in the multi-scale aggregation network layer is not limited to four. The more separated feature maps there are, the more features can be learned with richer receptive fields. The 3×3 standard convolution processing performed on each separated feature map can potentially accept the corresponding information of the previous separated feature map, so that each feature sub-map can increase its receptive field. Therefore, each multi-scale aggregation network layer can acquire different numbers and different receptive field combinations of features. Finally, the shorting operation of the input features can capture global and detailed information at a finer granular level.

[0067] The fine extraction layer in the feature extraction network G provided in this embodiment, by stacking multiple multi-scale aggregation network layers, can not only fully extract multi-scale feature information of object instances of various scales, but also reduce other redundant calculations, thereby further improving the overall network performance.

[0068] It should be noted that there is a one-to-one correspondence between high-level feature maps and multi-scale aggregation network layers. That is, each multi-scale aggregation network layer outputs the high-level feature map of the corresponding layer, and the resolution and channels of the high-level feature maps are different between the multiple layers.

[0069] Step 1023: After performing 1×1 convolution on the multi-layer high-level feature maps with different channels and different resolutions, a first fused feature map with multiple layers having the same channels but different resolutions is obtained.

[0070] In this embodiment, for application scenarios involving multi-scale objects with varying or even drastically different sizes, a targeted multi-scale aggregation network can effectively improve the representation capability of multi-scale features and increase the receptive field of different feature network layers to capture feature information of objects of different sizes, thereby improving the segmentation performance of subsequent multi-scale objects.

[0071] In one embodiment, please refer to Figure 5 The bounding box prediction network V includes a first prediction module, a second prediction module, a third prediction module, and an attention map extraction module. Please refer to [reference needed]. Figure 6 , Figure 6The flowchart for predicting the target bounding box output by the target bounding box prediction network V and the corresponding layer attention map includes steps 1031 to 1037.

[0072] Step 1031: Take each pixel in the first fused feature map with the same channel but different resolutions in multiple layers as a sample point to construct a sample set.

[0073] In this embodiment, sample points are categorized into positive samples, negative samples, and fuzzy samples. Pixels in the first fused feature map that fall within any of the marked target boxes are considered positive samples; pixels not located within any of the marked target boxes are considered negative samples; and pixels falling within at least two marked target boxes are considered fuzzy samples. It should be noted that all sample points in the sample set are input into the target box prediction network V for prediction. For predicted target boxes, the predicted target boxes of pixels corresponding to negative samples are not included in the training of the subsequent target box prediction network loss function. For fuzzy samples, the marked target box with the smallest area among the multiple marked target boxes in which the fuzzy sample falls is paired with the fuzzy sample for training the target box prediction network loss function.

[0074] All pixels in the sample set are input into the target box prediction network for prediction. Pixels in the sample set that meet the following conditions are used as training sample points. The predicted target boxes obtained by the training sample points and the corresponding labeled target boxes are used to train the target box prediction network loss function.

[0075] The training sample points satisfy the following condition: the distance information from each pixel in the first fusion feature map of each layer in the sample set to the four boundaries of the labeled target box after being mapped to the input image satisfies the following formula:

[0076] Min(j)≤max(l,t,r,b)<Max(j)

[0077] Where l, t, r, and b represent the distance information from each pixel in the j-th layer of the first fused feature map to the four boundaries of the target bounding box, Min(j) represents the minimum value of the distance information from each pixel in the j-th layer of the first feature map to the four boundaries of the target bounding box, and Max(j) represents the maximum value of the distance information from each pixel in the j-th layer of the first fused feature map to the four boundaries of the target bounding box; and both Min(j) and Max(j) increase with the increase of the layer number j. In this embodiment, the [Min(j), Max(j)) of each layer of the multi-layer first fused feature map from the lower layer feature map to the higher layer feature map is defined as [0, 64), [64, 128), [128, 256), ..., and so on.

[0078] For example, suppose a pixel in the first fusion feature map of layer j is mapped to the input image at coordinates (x, y). This pixel lies within a labeled bounding box, and its maximum distance from the four boundaries of the labeled bounding box is d, where 0 ≤ d < 64. Therefore, this pixel becomes the final training sample point, and the predicted bounding boxes obtained from these training sample points participate in the training. It should be noted that all pixels in the first fusion feature map of multiple layers are input into the bounding box prediction network as sample points from the sample set, but the final predicted bounding boxes used in training are those predicted by the training sample points that satisfy the above conditions.

[0079] Step 1032: Input the sample set into the first prediction module to obtain a first prediction feature sub-map with C channels and the same resolution as the first fusion feature map of the corresponding layer. The first prediction feature sub-map is used to characterize the class information of the predicted target box predicted by each pixel in the first fusion feature map of the corresponding layer, where C is the number of classes.

[0080] Step 1033: Input the sample set into the second prediction module to obtain a second prediction feature sub-map with multiple layers and the same resolution as the first fusion feature map of the corresponding layer. The second prediction feature sub-map is used to represent the distance information from each pixel in the first fusion feature map of the corresponding layer to the four boundaries of the predicted target box. The distance information from each pixel in the first fusion feature map to the four boundaries of the predicted target box is directly predicted by the target box prediction network V, and this predicted distance information is the distance information l corresponding to the position of each pixel in the first fusion feature map mapped to the input image F. * t * r * b * The specific mapping method will be explained in detail below, and will not be repeated here.

[0081] Step 1034: Input the sample set into the third prediction module to obtain a third prediction feature sub-map with one channel and the same resolution as the first fusion feature map of the corresponding layer. The third prediction feature sub-map is used to characterize the centrality of each pixel in the first fusion feature map of the corresponding layer. The centrality is the score of the position of each pixel in the first fusion feature map after being mapped to the input image F and located in the predicted target box.

[0082] Step 1035: Input the sample set into the attention map extraction module to obtain a multi-layer attention map a with a K×Q×Q channel dimension and the same resolution as the first fusion feature map of the corresponding layer; where K is the mask coefficient and Q is the length or width of the mask. In this embodiment, the complete shape of the attention map output by the attention map extraction module is [Bs,(K×Q×Q),H,W], where Bs is the number of input images F, H×W is the resolution of the attention map, K is the mask coefficient, and Q is the length or width of the mask. This embodiment designs the channel dimension of the attention map as a three-dimensional structure of (K×Q×Q) to better encode instance-level information in the first fusion feature map, such as the shape and pose of objects. In this embodiment, K is 4 and Q is 14.

[0083] Step 1036: Based on the multi-layer first prediction feature sub-map, multi-layer second prediction feature sub-map, and multi-layer third prediction feature sub-map, determine the category information, distance information to the four boundaries of the predicted target box, and centrality of each pixel in the multi-layer first fused feature map.

[0084] Step 1037: Based on the category information of each pixel in the multi-layer first fusion feature map, the distance information to the four boundaries of the predicted target box, and the centrality, obtain the position information, category information, and centrality of the predicted target box in each layer of the first fusion feature map, and then predict all the predicted target boxes.

[0085] Assume the height and width of the first fusion feature map in layer 1 are h and w, respectively, C is the number of categories for each pixel, and the stride of this first fusion feature map relative to the input image F is s. For example, if the input image F is 256×256 and the resolution of the first fusion feature map is 64×64, then the stride s is 256 / 64 = 4. The bounding box is denoted as Box, where Box = {x0, y0, x1, y1, c}, where {x0, y0} and {x1, y1} represent the coordinates of the top-left and bottom-right vertices of the bounding box Box, respectively, and c represents the category information (true category information) of the bounding box Box. Therefore, for the coordinates of each pixel in the first fusion feature map of layer 1 (f... x f y The following formula is used to map the input image F:

[0086]

[0087] Where (x, y) is a pixel in the first fusion feature map of layer (f x f y The coordinates mapped to the input image F; x f y ) represents the coordinates of each pixel in the first fused feature map of layer 1.

[0088] As mentioned above, the predicted target boxes output by the target box prediction network V need to be filtered before they can participate in subsequent training. The predicted target boxes predicted by the pixels corresponding to negative samples do not participate in the training of the loss function of the target box prediction network. For fuzzy samples, the smallest area of ​​the multiple labeled target boxes in which the fuzzy sample falls is selected and paired with the fuzzy sample for training of the loss function of the target box prediction network. Furthermore, the maximum value of the distance from the pixel of the first fusion feature map of each layer to the four boundaries of the labeled target box after the pixel is mapped to the original image position must be within a preset range. The filtering method of the predicted target boxes participating in training has been explained in detail above and will not be repeated here.

[0089] like Figure 7 As shown, the label values ​​l, t, r, and b corresponding to the distance information from the training pixels to the four boundaries of the predicted target box are calculated using the following formulas:

[0090] l=x-x0,t=y-y0,r=x1-x,b=y1-y

[0091] Where l, t, r, and b are the label values ​​corresponding to the distance information of the four boundaries of the predicted target box for each training pixel in the first fused feature map output by the target box prediction network V. The distance information from the training pixel output by the target box prediction network V to the four boundaries of the predicted target box is the predicted value, which is l. * t * r * b * The distance information from the training pixel to the four boundaries of the predicted target box obtained by the above formula is the label value, which is l, t, r, b respectively. During training, the target box prediction network loss function is trained by using the predicted value and label value of the same predicted target box.

[0092] In one embodiment, in step 104, based on the predicted target box's location information, category information, and centrality, as well as the location information and category information of the labeled target boxes in the labeled data, the target box prediction network loss function L is constructed according to the following formula. V :

[0093] L V =L class +L distance +L center

[0094] Among them, L class Represents the classification loss function. p n g represents the confidence level that the predicted target box belongs to category n. nThis represents the score for marking the target box as category n, where C is the total number of categories;

[0095] L distance This represents the distance prediction loss function. M represents the area of ​​the predicted target box, N represents the area of ​​the labeled target box, and M∩N represents the area of ​​overlap between the regions of the predicted target box and the labeled target box; where the processed labeled target box is the target box corresponding to the label values ​​l, t, r, b based on the distance information from the training pixels in the above-calculated sample set to the four boundaries of the predicted target box.

[0096] L center To represent the centrality loss function, l * t * r * b * This represents the distance information from each pixel in the first fused feature map to the four boundaries of the predicted target box.

[0097] In one embodiment, the masking network P includes a segmentation module S and a fusion module B. Please refer to [reference needed]. Figure 8 , Figure 8 The diagram below illustrates a structure of a segmentation module S, which includes two tasks. In step 105, the multi-layer first fusion feature map is input into the segmentation module S of the masking network P to obtain a predicted segmentation map and a predicted score map, including steps 1051 to 1054. Step 1051 is the first task of the segmentation module S, which outputs the predicted segmentation map f1 and the labeled binary map f1 corresponding to the input image F. t In the segmentation module loss function L S Under the guidance of the module, the pixel-level semantic binary classification task is completed; steps 1052 to 1054 are the second task of the segmentation module S, namely, outputting the predicted score map sm.

[0098] Step 1051: Perform multiple consecutive 3×3 convolution (C3x3) and 1×1 convolution (C1x1) processes on the first layer first fusion feature map α1 of the multi-layer first fusion feature map to obtain a predicted segmentation map f1 with 2 channels and the same resolution as the first layer first fusion feature map α1.

[0099] In this embodiment, the predicted segmentation map f1 is used to compare the labeled binary map corresponding to the input image F in the segmentation module loss function L. SGuided by this, a pixel-level semantic binary classification task is completed. This allows low-level features to more effectively capture the instance semantic information of foreground objects, which not only increases the accurate location information but also effectively suppresses the background. The labeled binary map is a binary pixel-level label map of the input image F after scale transformation (resize) operation, which is used to represent the foreground and background of pixels.

[0100] The segmentation module loss function L is constructed using the following formula. S :

[0101] L S =-f t logf1-(1-f t log(1-f1)

[0102] Among them, f t f1 represents the labeled binary image corresponding to the input image F after scaling, and f1 represents the predicted segmentation image.

[0103] Step 1052: Perform multiple consecutive 3×3 convolutions on the first fusion feature maps α2, α2, and α3 of the multi-layer first fusion feature maps, excluding the first-layer fusion feature map, to obtain the multi-layer output feature map.

[0104] Step 1053: Perform feature fusion on the multi-layer output feature map to obtain the second fused feature map.

[0105] Step 1054: After performing 3×3 convolution and upsampling on the second fused feature map, it is then processed by 1×1 convolution to obtain a prediction score map sm with K channels; where K is the value of the first dimension in the channel dimension of the multi-layer attention map, and in this embodiment, K is 4.

[0106] Please refer to Figure 9 In step 106, the multi-layer attention map, prediction score map and labeled target box are input into the fusion module B of the mask network P to obtain the instance prediction image, including steps 1061 to 1064.

[0107] Step 1061: Based on the step size of the input image relative to the predicted score map, scale the marked target box to map the marked target box to the predicted score map sm, and then process it through RoIAlign to obtain the target box cropped score map sz with a preset resolution Z*Z.

[0108] Step 1062: Select the target bounding boxes from the multi-layer attention map to obtain the attention vectors corresponding to the target bounding boxes.

[0109] Step 1063: Perform a concatenation operation on the attention vectors corresponding to the marked target boxes along the batch size dimension. Then, perform dimension transformation and interpolation on the result of the concatenation operation to obtain an attention map ad with a preset resolution Z*Z. In this embodiment, the preset resolution is 56x56. For example, assuming that there are 10 marked target boxes in the input image F, the shape of the resulting target box cropping fraction map with the preset resolution is [10,4,56,56].

[0110] Among these, the selection of multi-level attention maps based on the marked target boxes includes:

[0111] First, determine the center point of the labeled target box, remember its location, and calculate the maximum distance from the center point to the four boundaries of the labeled target box. Use this maximum distance to obtain the network layer corresponding to the attention map that meets the preset range (for example, if the maximum distance is 30, and 0 < 30 < 64, then the labeled target box corresponds to the first layer attention map). Then, map the labeled target box to the corresponding layer attention map to obtain the mapped target box attention map. Next, obtain the attention vector where the center point of the mapped target box attention map is located. In this way, obtain the corresponding attention vectors of all labeled target boxes and perform a concatenation operation on the batch processing dimension.

[0112] To make it easier to understand, let's take an example. The shape of the corresponding attention vector of the selected target box is [10, 4×14×14]. After performing the dimension transformation operation, the output shape is [10, 4, 14, 14]. After the interpolation operation, the output attention map ad shape is [10, 4, 56, 56].

[0113] Step 1064: Normalize the attention map ad with a preset resolution on the channel to obtain the normalized attention map sd.

[0114] Step 1065: Multiply the normalized attention map sd and the target box clipping fraction map sz at the element level, and sum the results of the multiplication on the channels to obtain the instance prediction image.

[0115] In one embodiment, step 108 involves constructing the loss function L of the fusion module B based on the instance prediction image and the binarized image corresponding to the marked target box in the labeled binary image of the input image F, using the following formula. R :

[0116] L R =-f g logf2-(1-f g log(1-f2)

[0117] Among them, f gf1 represents the binarized image corresponding to the marked target box in the labeled binary image of the input image F after scaling, and f2 represents the instance prediction image.

[0118] In one embodiment, in step 109, a total loss function L is constructed based on the target box prediction network loss function and the segmentation network loss function:

[0119] L = L V +L S +L R

[0120] Among them, L V L represents the loss function of the target bounding box prediction network. S L represents the segmentation module loss function. R This represents the loss function of the fusion module.

[0121] Based on the total loss function L constructed above, the multi-scale object instance segmentation model is trained to obtain the final model parameters, thus completing the training.

[0122] Based on the multi-scale object instance segmentation model trained above, please refer to... Figure 10 This embodiment provides a method for instance segmentation of multi-scale objects, including steps 201 to 207, which are described in detail below.

[0123] Step 201: Obtain an input image F that includes one or more object instances.

[0124] Step 202: Input the input image F into the feature extraction network G to obtain a first fusion feature map with multiple layers having the same channel but different resolutions.

[0125] Step 203: Input the first fusion feature maps with the same channels but different resolutions from the multi-layer first fusion feature maps into the target box prediction network V to obtain the location information, category information and centrality of the predicted target boxes in each layer's first fusion feature map, as well as the attention map corresponding to each layer's first fusion feature map, wherein the attention map has the same resolution as the first fusion feature map of the corresponding layer.

[0126] Step 204: Filter all predicted target boxes for each pixel in the first fusion feature map of each layer, and determine whether the predicted target box for each pixel in the first fusion feature map of each layer is a predicted instance target box; if it is a predicted instance target box, then take the corresponding pixel as the predicted pixel.

[0127] Step 205: Input the multi-layer first fusion feature map into the segmentation module S of the mask network P to obtain the predicted score map.

[0128] Step 206: Input the multi-layer attention map, the prediction score map, and the predicted instance target boxes into the fusion module B of the mask network P to obtain the instance prediction image; wherein, the instance prediction image contains the pixel segmentation results of the input image F with a preset resolution and the same number of predicted instance target boxes.

[0129] Step 207: Scale the pixel segmentation results of all predicted instance target boxes with preset resolution to the size of the corresponding predicted instance target boxes to obtain the pixel segmentation results of all predicted instance target boxes in the input image F. Then, the pixels in the input image F that do not fall within the predicted instance target boxes are defaulted to background pixels to obtain the final object instance segmentation result.

[0130] It should be noted that the feature extraction network G in this embodiment extracts features from the input image F in the same way as the training method provided in the above embodiments, and will not be repeated here. The differences between the training methods provided in the above embodiments and those in the target bounding box prediction network V and the masking network P are explained below.

[0131] In one embodiment, step 203 involves inputting multiple layers of first fused feature maps with the same channel but different resolutions into the target box prediction network V to obtain the location information, category information, and centrality of the predicted target boxes in each layer of the first fused feature map, including steps 2031 to 2035.

[0132] Step 2031: Input the first fusion feature maps with the same channels but different resolutions from the first fusion feature map of the corresponding layer into the first prediction module to obtain the first prediction feature sub-map with C channels and the same resolution as the first fusion feature map of the corresponding layer. The first prediction feature sub-map is used to characterize the category information of the predicted target box of each pixel in the first fusion feature map of the corresponding layer, where C is the number of categories.

[0133] Step 2032: Input the multi-layered first fusion feature maps with the same channels but different resolutions into the second prediction module to obtain multi-layered second prediction feature sub-maps with four channels and the same resolution as the corresponding layer's first fusion feature map. The second prediction feature sub-maps represent the distance information from each pixel in the corresponding layer's first fusion feature map to the four boundaries of the predicted target box. The distance information from each pixel in the first fusion feature map to the four boundaries of the predicted target box is directly predicted by the target box prediction network V. Furthermore, the distance information of the predicted target box's four boundaries is the distance information corresponding to the position of each pixel in the first fusion feature map mapped to the input image F. The specific mapping method has been described in detail in the above embodiments and will not be repeated here.

[0134] Step 2033: Input the first fusion feature maps with the same channels but different resolutions from the multi-layer first fusion feature map into the third prediction module to obtain the third prediction feature sub-map with one channel and the same resolution as the first fusion feature map of the corresponding layer. The third prediction feature sub-map is used to characterize the centrality of each pixel in the first fusion feature map of the corresponding layer.

[0135] Step 2034: Input the first fused feature maps with multiple layers having the same number of channels but different resolutions into the attention map extraction module to obtain multiple attention maps with K×Q×Q channel dimensions and the same resolution as the corresponding first fused feature maps. Here, K is the mask coefficient, and Q is the length or width of the mask.

[0136] Step 2035: Based on the multi-layer first prediction feature sub-map, multi-layer second prediction feature sub-map, and multi-layer third prediction feature sub-map, determine the category information of the predicted target box, the distance information to the four boundaries of the predicted target box, and the centrality of each pixel in the multi-layer first fused feature map.

[0137] Step 2036: Based on the category information of the predicted target box, the distance information to the four boundaries of the predicted target box and the centrality of each pixel in the multi-layer first fusion feature map, obtain the position information, category information and centrality of the predicted target box in each layer of the first fusion feature map, and then obtain the predicted target box in each layer of the first fusion feature map.

[0138] The predicted target boxes obtained by the target box prediction network V above still need to be filtered in the following way to obtain the predicted instance target boxes:

[0139] Determine whether the predicted bounding boxes in the first fusion feature maps of each layer meet the following three conditions:

[0140] The confidence level of the predicted target box in the first fusion feature map of each layer is greater than the first preset value;

[0141] The maximum length of the four boundaries of the predicted target box in the first fusion feature map of each layer is within a preset range. It should be noted that the length of the four boundaries of the predicted target box is the distance information after mapping the pixels in the first fusion feature map to the input image F.

[0142] The center prediction score obtained by multiplying the confidence level of each pixel in the first fusion feature map of each layer with the centrality of each pixel in the corresponding first fusion feature map is greater than the second preset value;

[0143] Finally, when all three conditions are met, Non-Maximum Suppression (NMS) is used to filter the predicted bounding boxes. The predicted bounding boxes of the filtered pixels are the predicted instance bounding boxes.

[0144] The maximum length of the four boundaries of the predicted target box in the first fusion feature map of each layer is within a preset range, including:

[0145] After mapping each pixel in the multi-layer first fusion feature map to its position in the input image, the distance information to the four boundaries of the predicted target box satisfies the following formula:

[0146] Min(j)≤max(l * ,t*,r*,b*)<Max(j)

[0147] Among them, l * t * r * b * Let Min(j) represent the distance information of each pixel in the first fusion feature map of the j-th layer after it is mapped to the input image and then to the four boundaries of the corresponding predicted target box. Let Max(j) represent the minimum value of the distance information of each pixel in the first fusion feature map of the j-th layer after it is mapped to the input image and then to the four boundaries of the corresponding predicted target box. Let Max(j) represent the maximum value of the distance information of each pixel in the first fusion feature map of the j-th layer after it is mapped to the input image and then to the four boundaries of the corresponding predicted target box. Both Min(j) and Max(j) increase as the number of layers j increases.

[0148] In one embodiment, step 206 involves inputting the multi-layer attention map, the prediction score map, and the predicted instance target box into the fusion module B of the mask network P to obtain the instance prediction image, including steps 2051 to 2056.

[0149] Step 2051: Based on the position of the predicted pixel in the first fusion feature map of each layer, obtain the attention vector of each predicted pixel in the corresponding layer attention map of the first fusion feature map of each layer.

[0150] Step 2052: Based on the attention vectors at the corresponding positions of each predicted pixel in the corresponding layer attention map, obtain the attention vectors corresponding to the target boxes of all predicted instances.

[0151] Step 2053: Perform a concatenation operation on the batch size dimension for the attention vectors corresponding to the target boxes of all predicted instances, and then perform dimension transformation and interpolation to obtain an attention map with a preset resolution.

[0152] Step 2054: Map the predicted instance bounding boxes to the predicted score map, and then process them using RoIAlign to obtain a target box clipping score map with a preset resolution. Assume that the shape of the attention vector corresponding to the predicted instance bounding box is [10, 4×14×14], the shape after the dimension transformation operation is [10, 4, 14, 14], and the shape of the attention map with the preset resolution after interpolation is [10, 4, 56, 56].

[0153] Step 2055: Normalize the attention map with a preset resolution on the channel to obtain a normalized attention map.

[0154] Step 2056: Multiply the normalized attention map and the target box clipping fraction map at the element level, and sum the results of the multiplication on the channels to obtain the instance prediction image.

[0155] The multi-scale object instance segmentation method provided in this invention firstly employs a more efficient and compact feature extraction network G designed specifically for the characteristics of multi-scale object data. This network is capable of extracting rich feature information. Furthermore, the unique fusion method of low-level feature scores and high-level feature attention maps enables accurate prediction of multi-scale target objects, significantly improving segmentation quality. Secondly, the target bounding box is directly predicted from the extracted first fused feature map, eliminating the need to set pre-set anchor box parameters. This method has a low technical threshold, requires minimal professional knowledge, and can be used for different tasks and object data without adaptation, demonstrating high versatility. Finally, the specially designed feature extraction network G greatly reduces the number of parameters and redundant computations in the multi-scale object segmentation model. The lightweight structure enables the inference speed of the multi-scale object segmentation model to reach real-time levels, saving detection time and achieving cost reduction and efficiency improvement for manufacturing enterprises.

[0156] Please refer to Figure 11 , Figure 11 This diagram illustrates the effect of inputting the input image F into a multi-scale object instance segmentation model for instance segmentation. Figure 11 The image on the left is the input image F. Figure 11 The image on the right is the object instance segmentation result output by the multi-scale object instance segmentation model.

[0157] Please refer to Figure 12 This invention also provides a multi-scale object instance segmentation device, which includes an image acquisition device 301 and a processor 302. The components can be connected via a bus or other methods.

[0158] The image acquisition device 301 is used to acquire an input image F including at least one or more object instances; the processor 302 is used to execute the multi-scale object instance segmentation method provided in any of the above embodiments.

[0159] The image acquisition device 301 can be, for example, a vision sensor or other device capable of acquiring image data. The processor 302 can be an integrated circuit chip with signal processing capabilities. The processor 302 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0160] Understandable. Figure 12 The structure shown is for illustrative purposes only and may include more... Figure 12 The more or fewer components shown, or having the same Figure 12 Different configurations. Figure 12 The components shown can be implemented in hardware and / or software. For example, they may also include a memory (not shown) for storing acquired image data, and a computer program that can be executed by processor 302 to implement the technical solution of the multi-scale object instance segmentation method provided in any of the above method embodiments. The memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0161] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0162] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A method for training a multi-scale object instance segmentation model, characterized in that, The multi-scale object instance segmentation model includes a feature extraction network, a bounding box prediction network, and a masking network, wherein the masking network includes a segmentation module and a fusion module, and the training method includes: The method involves acquiring an input image containing one or more object instances and annotation data, wherein the annotation data includes the location information and category information of the marked target boxes in the input image and the corresponding binary image of the input image. The input image is input into the feature extraction network to obtain a first fusion feature map with multiple layers having the same channel but different resolutions; The first fusion feature maps with the same channels but different resolutions are input into the target box prediction network to obtain the position information, category information, and centrality of the predicted target boxes in each layer of the first fusion feature map, as well as the attention map corresponding to each layer of the first fusion feature map. The attention map of each layer has the same resolution as the first fusion feature map of the corresponding layer. The centrality is used to characterize the centrality of each pixel in the first fusion feature map in the corresponding predicted target box. Based on the predicted target box's location information, category information, and centrality, as well as the labeled target box's location information and category information in the labeled data, a target box prediction network loss function is constructed. The first fusion feature map of the multilayer is input into the segmentation module of the mask network to obtain a predicted segmentation map and a predicted score map. The predicted segmentation map has 2 channels and has the same resolution as the first layer first fusion feature map of the first fusion feature map of the multilayer. The predicted score map has 4 channels and has the same resolution as the first layer first fusion feature map of the first fusion feature map of the multilayer. Based on the step size of the input image relative to the predicted score map, the marked target box is scaled to map the marked target box to the predicted score map, and then processed by RoIAlign to obtain a target box cropped score map with a preset resolution. The attention vectors corresponding to the marked target boxes are obtained by selecting from the multi-layered attention maps based on the marked target boxes; The attention vectors corresponding to the marked target boxes are concatenated in the batch size dimension. Then, the result of the concatenation operation is subjected to dimension transformation and interpolation to obtain an attention map with a preset resolution. The attention map with the preset resolution is normalized on the channel to obtain a normalized attention map; The normalized attention map and the target box clipping fraction map are multiplied at the element level, and the result of the multiplication is summed on the channel to obtain the instance prediction image. Based on the predicted segmentation map and the labeled binary map corresponding to the scale-transformed input image, a segmentation module loss function is constructed; wherein, the labeled binary map corresponding to the scale-transformed input image has the same resolution as the predicted segmentation map; Based on the instance prediction image and the binarized image corresponding to the marked target box in the labeled binary image of the scale-transformed input image, a fusion module loss function is constructed; wherein, the binarized image and the instance prediction image have the same resolution; Based on the target bounding box prediction network loss function, the segmentation module loss function, and the fusion module loss function, a total loss function is constructed, and the multi-scale object instance segmentation model is trained based on the total loss function to obtain the corresponding model parameters.

2. The method of claim 1, wherein, The feature extraction network includes a coarse extraction layer and a fine extraction layer; inputting the input image into the feature extraction network yields a first fused feature map with multiple layers having the same channel but different resolutions, including: The input image is processed twice by the coarse extraction layer to obtain the initial feature map; The initial feature map is processed by multiple multi-scale aggregation network layers in the fine extraction layer to obtain multiple high-level feature maps with different channels and resolutions. After performing 1×1 convolution on the multi-layer high-level feature maps with different channels and different resolutions, a first fused feature map with multiple layers having the same channels but different resolutions is obtained.

3. The method of claim 2, wherein, The initial feature map is input into a fine extraction layer and processed through multiple multi-scale aggregation network layers, including: If the current multi-scale aggregation network layer processing is the first multi-scale aggregation network layer processing, the first multi-scale aggregation network layer processing shall be performed in the following manner: The initial feature map is subjected to 1×1 convolution and channel averaging separation to obtain multiple separated feature maps; The first separate feature map among multiple separate feature maps is taken as the first feature sub-map. The second separate feature map among multiple separate feature maps is convolved to obtain the second feature sub-map. The i-th separate feature map and the (i-1)-th feature sub-map among multiple separate feature maps are subjected to Hadamard addition and then convolved to obtain the i-th feature sub-map, so as to obtain multiple feature sub-maps, where i is an integer greater than or equal to 3. After concatenating multiple feature sub-images by channel, a 1×1 convolution is performed to obtain the first feature image; The initial feature map and the first feature map are subjected to Hadamard addition to obtain the first high-level feature map; If the current multi-scale aggregation network layer processing is not the first multi-scale aggregation network layer processing, perform a multi-scale aggregation network layer processing as follows: The feature map output from the previous multi-scale aggregation network layer is processed by 1×1 convolution and channel averaging to obtain multiple separated feature maps. The first separate feature map among multiple separate feature maps is taken as the first feature sub-map. The second separate feature map among multiple separate feature maps is convolved to obtain the second feature sub-map. The ith separate feature map and the (i-1)th feature sub-map among multiple separate feature maps are subjected to Hadamard addition and then convolved to obtain the ith feature sub-map, so as to obtain multiple feature sub-maps, where i is an integer greater than or equal to 3. After concatenating multiple feature sub-images by channel, a 1×1 convolution is performed to obtain the first feature image; The Hadamard addition operation is performed on the feature map output by the previous multi-scale aggregation network layer and the first feature map to obtain a higher-level feature map.

4. The method of claim 1, wherein, The target bounding box prediction network includes: a first prediction module, a second prediction module, a third prediction module, and an attention map extraction module; the multi-layered first fused feature maps with the same channel but different resolutions are input into the target bounding box prediction network to obtain the location information, category information, and centrality of the predicted target bounding boxes in each layer of the feature maps, as well as the attention maps corresponding to each layer of the feature maps, including: Each pixel in the first fused feature map of the multi-layered system with the same channel but different resolution is used as a sample point to construct a sample set. The sample set is input into the first prediction module to obtain a first prediction feature sub-map with multiple layers and the same resolution as the first fusion feature map of the corresponding layer, wherein the first prediction feature sub-map is used to characterize the category information of the predicted target box of each pixel in the first fusion feature map of the corresponding layer, and C is the number of categories. The sample set is input into the second prediction module to obtain a second prediction feature sub-map with four channels and the same resolution as the first fusion feature map of the corresponding layer. The second prediction feature sub-map is used to characterize the distance information of each pixel in the first fusion feature map of the corresponding layer to the four boundaries of the prediction target box. The sample set is input into the third prediction module to obtain a third prediction feature sub-map with one channel and the same resolution as the first fusion feature map of the corresponding layer. The third prediction feature sub-map is used to characterize the centrality of each pixel in the first fusion feature map of the corresponding layer. The sample set is input into the attention map extraction module to obtain a multi-layer attention map with K×Q×Q channel dimensions and the same resolution as the first fused feature map of the corresponding layer; where K is the mask coefficient and Q is the length or width of the mask; Based on the first multi-layer prediction feature sub-map, the second multi-layer prediction feature sub-map, and the third multi-layer prediction feature sub-map, determine the category information, distance information to the four boundaries of the predicted target box, and centrality of each pixel in the first multi-layer fused feature map. Based on the category information of each pixel in the first fusion feature map of multiple layers, the distance information to the four boundaries of the predicted target box, and the centrality, the position information, category information, and centrality of the predicted target box in the first fusion feature map of each layer are obtained.

5. The method of claim 4, wherein, The step of constructing a sample set by treating each pixel in the first fused feature map with the same channel but different resolutions across multiple layers as a sample includes: Pixels located in a target box in the first fused feature map of the multi-layered system with the same channel but different resolutions are taken as positive samples. Pixels in the first fused feature map of the multi-layered system with the same channel but different resolution that are not located in any of the marked target boxes are taken as negative samples. Pixels that fall simultaneously in at least two labeled target boxes in the first fused feature map of multiple layers with the same channel but different resolutions are used as blurred samples. Based on the positive samples, negative samples, and fuzzy samples, a sample set is obtained.

6. The method of claim 5, wherein, Based on the positive samples, negative samples, and fuzzy samples, the resulting sample set includes: Determine whether each pixel in the first fused feature map of the multiple layers meets the following conditions: The distance information from each pixel in the first fused feature map of the multilayer to the four boundaries of the marked target box after mapping to the input image position satisfies the following formula: ； wherein, , , , represents distance information of each pixel point in the jth layer first fusion feature map to the four boundaries of the marked target frame after being mapped to the input image, represents the minimum value of the distance information of each pixel point in the preset jth layer first fusion feature map to the four boundaries of the marked target frame after being mapped to the input image, represents the maximum value of the distance information of each pixel point in the preset jth layer first fusion feature map to the four boundaries of the marked target frame after being mapped to the input image; and and both increase with the increase of the layer number j. Pixels in the sample set that meet the above conditions are used as training sample points. All sample points in the sample set are input into the target box prediction network for prediction. The predicted target boxes obtained by the training pixels in the sample set are compared with the corresponding labeled target boxes to train the target box prediction network loss function.

7. The method as described in claim 1, characterized in that, Based on the predicted target bounding box's location information, category information, and centrality, as well as the labeled target bounding boxes' location information and category information in the labeled data, the target bounding box prediction network loss function is constructed as follows: The loss function of the target box prediction network is constructed using the following formula. : ； in, Represents the classification loss function. , This indicates that the predicted target box is a category. confidence level This represents the score for marking the target box as category n, where C is the total number of categories; This represents the distance prediction loss function. M represents the area of ​​the predicted bounding box, and N represents the area of ​​the labeled bounding box. This represents the area where the predicted target box and the processed labeled target box overlap; wherein the processed labeled target box is obtained by processing the vertex coordinates of the labeled target box and the coordinates of the training pixels in the sample set corresponding to the predicted target box; Represents the centrality loss function. , * , * , * , * This represents the distance information from each pixel in the first fused feature map to the four boundaries of the predicted target box.

8. The method as described in claim 1, characterized in that, The first fused feature map from multiple layers is input into the segmentation module of the masking network to obtain a predicted segmentation map and a predicted score map, including: The first layer of the first fusion feature map in the multi-layer first fusion feature map is subjected to multiple consecutive 3×3 convolution and 1×1 convolution processes to obtain a predicted segmentation map with 2 channels and the same resolution as the first layer of the first fusion feature map. Perform multiple consecutive 3×3 convolutions on the first fusion feature maps of the multi-layer first fusion feature maps, except for the first layer fusion feature map, to obtain the multi-layer output feature maps; The multi-layer output feature map is fused to obtain a second fused feature map; After performing 3×3 convolution and upsampling on the second fused feature map, it is then processed by 1×1 convolution to obtain a prediction score map with K channels; where K is the value of the first dimension in the channel dimension of the multi-layer attention map.

9. The method as described in claim 1, characterized in that, Based on the predicted segmentation map and the labeled binary map corresponding to the scaled input image, a segmentation module loss function is constructed. The loss function of the segmentation module is constructed using the following formula. : ； in, This represents the labeled binary image corresponding to the scaled input image. This represents the predicted segmentation map.

10. The method as described in claim 1, characterized in that, Based on the instance prediction image and the binarized image corresponding to the marked target box in the marked binary image of the input image, a fusion module loss function is constructed. The loss function of the fusion module is constructed using the following formula. : ； in, This represents the binarized image corresponding to the marked target box in the labeled binary image of the input image after scaling. This represents the predicted image for the instance.

11. The method as described in claim 1, characterized in that, Based on the target bounding box prediction network loss function, the segmentation module loss function, and the fusion module loss function, the total loss function is constructed as follows: The total loss function is constructed using the following formula. : ； in, This represents the loss function of the target box prediction network. This represents the loss function of the segmentation module. This represents the loss function of the fusion module.

12. A method for instance segmentation of multi-scale objects, characterized in that, include: Obtain an input image that includes one or more object instances; The input image is input into a feature extraction network to obtain a first fusion feature map with multiple layers having the same channel but different resolutions. The first fusion feature maps with the same channels but different resolutions are input into the target box prediction network to obtain the position information, category information and centrality of the predicted target boxes in the first fusion feature maps of each layer, as well as the attention map corresponding to the first fusion feature map of each layer, wherein the attention map of each layer has the same resolution as the first fusion feature map of the corresponding layer. For each pixel in the first fusion feature map of each layer, all predicted target boxes are filtered to determine whether the predicted target box for each pixel in the first fusion feature map of each layer is a predicted instance target box; if it is a predicted instance target box, the corresponding pixel is taken as the predicted pixel. The first fused feature map from multiple layers is input into the segmentation module of the mask network to obtain the predicted score map; Based on the position of the predicted pixel in the first fusion feature map of each layer, obtain the attention vector corresponding to each predicted pixel in the attention map of the corresponding layer of the first fusion feature map of each layer. Based on the attention vectors at the corresponding positions of each predicted pixel in the corresponding layer attention map, the attention vectors corresponding to the target boxes of all predicted instances are obtained. The attention vectors corresponding to the target boxes of all predicted instances are concatenated along the batch size dimension, and then dimension transformation and interpolation are performed to obtain an attention map with a preset resolution. Based on the step size of the input image relative to the prediction score map, the predicted instance target box is scaled and mapped to the prediction score map. Then, it is processed by RoIAlign to obtain a target box cropping score map with a preset resolution. The attention map with the preset resolution is normalized on the channel to obtain a normalized attention map; The normalized attention map and the target box clipping fraction map are multiplied at the element level, and the result of the multiplication is summed on the channel to obtain an instance prediction image. The instance prediction image contains the pixel segmentation results of the input image with a preset resolution and the same number of predicted instance target boxes as the predicted instance target boxes. The pixel segmentation results of all predicted instance target boxes with preset resolutions are scaled to the size of the corresponding predicted instance target boxes to obtain the pixel segmentation results of all predicted instance target boxes in the input image. Pixels in the input image that do not fall within the predicted instance target boxes are defaulted to background pixels to obtain the final object instance segmentation result.

13. The method as described in claim 12, characterized in that, The target bounding box prediction network includes: a first prediction module, a second prediction module, a third prediction module, and an attention map extraction module; the multi-layered first fused feature maps with the same channel but different resolutions are input into the target bounding box prediction network to obtain the location information, category information, and centrality of the predicted target bounding boxes in each layer of the first fused feature maps, and the attention maps corresponding to each layer of the first fused feature maps include: The first fusion feature maps with the same channels but different resolutions are input into the first prediction module to obtain a first prediction feature sub-map with C channels and the same resolution as the first fusion feature map of the corresponding layer. The first prediction feature sub-map is used to characterize the category information of the predicted target box of each pixel in the first fusion feature map of the corresponding layer, where C is the number of categories. The first fusion feature map with the same channels but different resolutions in the multi-layered system is input into the second prediction module to obtain a second prediction feature sub-map with four channels in the multi-layered system and the same resolution as the first fusion feature map of the corresponding layer. The second prediction feature sub-map is used to characterize the distance information of each pixel in the first fusion feature map of the corresponding layer to the four boundaries of the prediction target box. The first fusion feature maps with the same channels but different resolutions are input into the third prediction module to obtain a third prediction feature sub-map with one channel and the same resolution as the first fusion feature map of the corresponding layer. The third prediction feature sub-map is used to characterize the centrality of each pixel in the first fusion feature map of the corresponding layer. The first fused feature maps with the same channels but different resolutions are input into the attention map extraction module to obtain an attention map with multiple channels of K×Q×Q and the same resolution as the first fused feature map of the corresponding layer; where K is the mask coefficient and Q is the length or width of the mask. Based on the first multi-layer prediction feature sub-map, the second multi-layer prediction feature sub-map, and the third multi-layer prediction feature sub-map, determine the category information, distance information to the four boundaries of the predicted target box, and centrality of each pixel in the first multi-layer fused feature map. Based on the category information of each pixel in the first fusion feature map of multiple layers, the distance information to the four boundaries of the predicted target box, and the centrality, the position information, category information, and centrality of the predicted target box in the first fusion feature map of each layer are obtained.

14. The method as described in claim 12, characterized in that, Determining whether the predicted bounding box of each pixel in the first fused feature map of each layer is a predicted instance bounding box includes: Determine whether the predicted bounding box for each pixel in the first fused feature map of each layer meets the following conditions: The confidence level of the predicted target box for each pixel in the first fused feature map of each layer is greater than the first preset value; The maximum length of the four boundaries of the predicted target box predicted by each pixel in the first fused feature map of each layer is within a preset range. The center prediction score obtained by multiplying the confidence level of each pixel in the first fusion feature map of each layer with the centrality of each pixel in the corresponding first fusion feature map is greater than the second preset value. When all the above conditions are met, non-maximum suppression is then used to filter the predicted target boxes of each pixel. The predicted target boxes of the filtered pixels are the predicted instance target boxes.

15. The method as described in claim 14, characterized in that, The maximum length of the four boundaries of the predicted bounding box for each pixel in the first fused feature map of each layer is within a preset range, including: The distance information from each pixel in the first fused feature map of the multilayer to the four boundaries of the predicted target box after mapping to the input image position satisfies the following formula: ； in, , , , This represents the distance information of each pixel in the first fusion feature map of the j-th layer to the four boundaries of the corresponding predicted target box after being mapped to the input image. This represents the minimum distance information from each pixel in the preset j-th layer of the first fusion feature layer to the four boundaries of the corresponding predicted target box after mapping to the input image. This represents the maximum value of the distance information from each pixel in the preset j-th layer first fusion feature map to the four boundaries of the corresponding predicted target box after mapping to the input image; and and Both increase as the number of layers j increases.

16. A multi-scale object instance segmentation device, characterized in that, include: Image acquisition device, used to acquire input images; A processor for executing the instance segmentation method for multi-scale objects as described in any one of claims 12 to 15.

17. A computer-readable storage medium, characterized in that, The medium stores a program that can be executed by a processor to implement the method as described in any one of claims 1 to 15.

Images