Instance segmentation and instance segmentation network training method, device, medium, and equipment

Abstract

The disclosed embodiments disclose a method and apparatus, medium, and device for instance segmentation and instance segmentation network training, wherein the method comprises: utilizing a first network branch to perform feature extraction on an image to be processed, obtaining multiple image features output by multiple network layers in the first network branch; utilizing an encoder to encode the multiple image features to obtain pixel-level features; utilizing an encoder and a decoder to process a first image feature among the multiple image features to obtain at least one object embedding included in the image to be processed; and determining a mask and category corresponding to at least one object in the image to be processed based on the pixel-level features and the at least one object embedding. The disclosed embodiments extract object embeddings, through which the category of at least one object in the image can be directly determined, thereby directly obtaining instance segmentation results and achieving end-to-end image instance segmentation.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology, and in particular to an instance segmentation and instance segmentation network training method, apparatus, medium, and device. Background Art

[0002] Image instance segmentation is fundamental to research in applications such as autonomous driving. Unlike semantic segmentation, which segments objects along the categorical dimension, instance segmentation distinguishes between different objects of the same class, expanding the algorithm's application scenarios and content richness. In some complex and detailed scenarios, instance segmentation also produces more robust results than semantic segmentation algorithms. Summary of the Invention

[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a method and apparatus, medium, and device for instance segmentation and instance segmentation network training.

[0004] According to one aspect of an embodiment of the present disclosure, a method for instance segmentation is provided, including:

[0005] Performing feature extraction on the image to be processed using the first network branch to obtain a plurality of image features output by a plurality of network layers in the first network branch; wherein the plurality of network layers correspond to the plurality of image features in a one-to-one manner;

[0006] Encoding the plurality of image features using an encoder to obtain pixel-level features;

[0007] Processing a first image feature of the plurality of image features using the encoder and the decoder to obtain at least one object embedding included in the image to be processed;

[0008] Based on the pixel-level features and the at least one object embedding, a mask and a category corresponding to at least one object in the image to be processed are determined.

[0009] According to another aspect of an embodiment of the present disclosure, a method for training an instance segmentation network is provided, comprising:

[0010] Inputting a training image into an instance segmentation network to obtain a predicted category and a predicted mask corresponding to at least one predicted object in the training image;

[0011] determining a first loss based on the at least one predicted category and at least one true category corresponding to the training image;

[0012] determining a second loss based on the at least one predicted mask and at least one true mask corresponding to the training image;

[0013] The training of the instance segmentation network is supervised based on the first loss and the second loss.

[0014] According to another aspect of an embodiment of the present disclosure, there is provided an instance segmentation apparatus, including:

[0015] a feature extraction module, configured to perform feature extraction on the image to be processed using the first network branch, and obtain a plurality of image features output by a plurality of network layers in the first network branch; wherein the plurality of network layers correspond to the plurality of image features in a one-to-one manner;

[0016] A feature encoding module, configured to encode the multiple image features obtained by the feature extraction module using an encoder to obtain pixel-level features;

[0017] a feature decoding module, configured to process a first image feature among the plurality of image features obtained by the feature extraction module using the encoder and the decoder to obtain at least one object embedding included in the image to be processed;

[0018] An instance segmentation module is configured to determine a mask and a category corresponding to at least one object in the image to be processed based on the pixel-level features obtained by the feature encoding module and the at least one object embedding obtained by the feature decoding module.

[0019] According to another aspect of the present disclosure, there is provided an instance segmentation network training apparatus, comprising:

[0020] a network prediction module, configured to input a training image into an instance segmentation network to obtain a predicted category and a predicted mask corresponding to at least one predicted object in the training image;

[0021] a first loss determination module, configured to determine a first loss based on at least one predicted category obtained by the network prediction module and at least one true category corresponding to the training image;

[0022] a second loss determination module, configured to determine a second loss based on at least one predicted mask obtained by the network prediction module and at least one true mask corresponding to the training image;

[0023] A network training module is configured to supervise the training of the instance segmentation network based on the first loss determined by the first loss determination module and the second loss determined by the second loss determination module.

[0024] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above embodiments.

[0025] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device including:

[0026] processor;

[0027] a memory for storing instructions executable by the processor;

[0028] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of the above embodiments.

[0029] The above-mentioned embodiments of the present disclosure provide instance segmentation and instance segmentation network training methods, devices, media, and equipment. The embodiments of the present disclosure extract object embedding, which can directly determine the category of at least one object in the image, and use the object embedding and pixel-level features to obtain the mask of at least one object in the image, thereby directly obtaining the instance segmentation result and realizing end-to-end image instance segmentation. Through end-to-end image instance segmentation, the input image to be processed can be directly obtained as the output instance segmentation result; avoiding the dimensionality disaster caused by non-end-to-end image processing due to the large number of image pixels and high data dimension.

[0030] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other purposes, features, and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0032] Figure 1 2 is a schematic diagram of the structure of an instance segmentation network provided by an exemplary embodiment of the present disclosure.

[0033] Figure 2 It is a flowchart of an instance segmentation method provided by an exemplary embodiment of the present disclosure.

[0034] Figure 3 This disclosure Figure 2 A flow chart of step 202 in the embodiment shown is shown.

[0035] Figure 4 This disclosure Figure 3 A flow chart of step 2022 in the embodiment shown is shown.

[0036] Figure 5 This disclosure Figure 2 A flow chart of step 204 in the embodiment shown is shown.

[0037] Figure 64 is a flowchart of an example segmentation network training method provided by an exemplary embodiment of the present disclosure.

[0038] Figure 7 3 is a schematic diagram of the structure of an instance segmentation device provided by an exemplary embodiment of the present disclosure.

[0039] Figure 8 3 is a schematic structural diagram of an instance segmentation apparatus provided by another exemplary embodiment of the present disclosure.

[0040] Figure 9 2 is a schematic diagram of the structure of an example segmentation network training device provided by an exemplary embodiment of the present disclosure.

[0041] Figure 10 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0043] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0044] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0045] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0046] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0047] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0048] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0049] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0050] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0051] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0052] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0053] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0054] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0055] Application Overview

[0056] In the process of implementing the present disclosure, the inventors discovered that existing image instance segmentation usually adopts a single-stage segmentation method (Single Shot Instance Segmentation), but this method has at least the following problems: it requires detection assistance and requires redundancy removal operations.

[0057] Exemplary Systems

[0058] Figure 1 Schematic diagram of the structure of an instance segmentation network provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, it includes a first network branch 101, an encoder 102 and a decoder 103;

[0059] Among them, the first network branch 101 extracts features of the input image and outputs multiple image features respectively through multiple network layers included in the first network branch 101. The multiple image features are multi-level outputs of the first network branch. The multiple image features include shallow network features and deep network features, wherein each image feature corresponds to a network layer.

[0060] The encoder 102 in this embodiment can be a transformer encoder, which encodes the first image feature with a smaller resolution (for example, the smallest resolution) extracted by the first network branch 101 through the encoder 102. The first image feature represents a deep feature (the feature output by the network layer with a lower order in the first network branch is a deep feature. For example, the first network branch includes the first, second, third, fourth and fifth network layers connected in sequence, wherein the fourth and fifth network layers belong to the network layers with a lower order), and the encoded second image feature is fused (for example, added or spliced, etc.) with the first image feature to obtain the third image feature. The deep features of the network are enriched by fusing the first image feature and the second image feature. Preferably, since the resolution of the third image feature is relatively small, the third image feature and the image features of other resolutions output by the first network branch 101 can be scaled to the same size (for example, by utilizing the PPM module in the related technology) and further fused (fuse, for example, addition, etc.), thereby obtaining a multi-scale pixel-level feature that fuses the multi-resolution features, wherein each of the multiple points included in the pixel-level feature represents the detail feature of the point (for example, texture, local detail feature, etc.).

[0061] The decoder 103 in this embodiment can be a transformer decoder, which encodes and decodes the first image feature by combining the encoder and the decoder to obtain a sparse object embedding. Each object embedding represents a predicted object, so the category of the object and the position of the center point of the object in the image can be directly predicted.

[0062] A sparse prediction mask can be obtained by performing matrix correlation operation on the pixel-level features output by the encoder 102 and the object embedding output by the decoder 103.

[0063] Object embedding means converting large sparse vectors into a low-dimensional space that preserves semantic relationships. Each object consists of many pixels and has a unified representation called object embedding. That is, an object embedding refers to multiple pixels corresponding to a predicted object. By determining the multiple pixels corresponding to the predicted object, the position of the corresponding center point in the image can be determined.

[0064] Based on the category output by the encoder 102 and the prediction mask output by the decoder 103, an instance segmentation result corresponding to the image can be constructed.

[0065] This embodiment utilizes the encoder and decoder's ability to extract object features, and each object embedding output by the decoder corresponds to a prediction result. Because there are no multiple prediction results (multiple prediction results include redundant results, requiring non-maximum suppression to determine one of them), there is no need for non-maximum suppression (NMS) to implement redundancy removal, resulting in an instance segmentation network model that achieves a better trade-off between processing speed and accuracy. At the same time, in the training process of the instance segmentation network model provided by this embodiment, the training image only needs to know at least one true category and at least one true mask corresponding to at least one true category, without the need for redundant artificial priors, such as the complex processes of anchor points or positive and negative sample allocation in instance segmentation training in the prior art, making it more concise and efficient.

[0066] Exemplary Methods

[0067] Figure 2 FIG is a flow chart of an example segmentation method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 2 As shown, the following steps are included:

[0068] Step 201: Use the first network branch to perform feature extraction on the image to be processed to obtain multiple image features output by multiple network layers in the first network branch.

[0069] Among them, there is a one-to-one correspondence between multiple network layers and multiple image features.

[0070] Optionally, each network layer outputs different image features respectively, and the output multiple image features include features of the shallow and deep layers of the network. Different image features have different resolutions because they are output through different network layers. The resolution of shallow features is higher, and the resolution of deep features is lower.

[0071] Step 202: Encode the multiple image features using an encoder to obtain pixel-level features.

[0072] The encoder in this embodiment can refer to Figure 1 The encoder 102 in the provided instance segmentation network obtains pixel-level features by encoding multiple image features.

[0073] Step 203: Process a first image feature among the multiple image features using an encoder and a decoder to obtain at least one object embedding included in the image to be processed.

[0074] Optionally, the first image feature is a deep feature with lower resolution, for example, a deep feature with the lowest resolution output by the last layer of the first network branch.

[0075] The encoder in this embodiment can refer to Figure 1 The decoder 103 in the provided network result is understood, where embedding means converting a large sparse vector into a low-dimensional space that retains semantic relationships, and segmentation is to predict the category to which a pixel in the image belongs. The feature of each pixel is called a pixel-level feature; and each object includes many pixels, which form a unified representation called object embedding. Each predicted object corresponds to an embedding. There is no situation in the prior art where multiple candidate boxes are predicted for each object, which reduces redundant predictions. In the absence of redundant predictions, the non-maximum suppression step is no longer required, and the instance segmentation speed is faster.

[0076] Step 204 : Determine a mask and a category corresponding to at least one object in the image to be processed based on the pixel-level features and the at least one object embedding.

[0077] Optionally, a sparse prediction mask is obtained by utilizing at least one object embedding and pixel-level features, thereby directly obtaining an instance segmentation result without the need for steps such as non-maximum suppression post-processing, thereby achieving end-to-end image instance segmentation.

[0078] The instance segmentation method provided by the above-mentioned embodiment of the present disclosure extracts object embedding, and the category of at least one object in the image can be directly determined through the object embedding, and the object embedding is used to interact with pixel-level features to obtain a mask of at least one object in the image, thereby directly obtaining the instance segmentation result and realizing end-to-end image instance segmentation. Through end-to-end image instance segmentation, the input image to be processed is realized, and the output instance segmentation result is directly obtained; and the dimensionality disaster caused by non-end-to-end image processing due to the large number of image pixels and high data dimension is avoided.

[0079] like Figure 3 As shown in the above Figure 2 Based on the embodiment shown, step 202 may include the following steps:

[0080] Step 2021: Encode the first image feature among the multiple image features using an encoder to obtain a second image feature.

[0081] In this embodiment, the first image feature is a deep feature with lower resolution among multiple image features, for example, the image feature with the lowest resolution output by the last network layer of the first network branch.

[0082] Step 2022: Fusing the second image feature with multiple image features to obtain pixel-level features.

[0083] In this embodiment, since the resolution of the first image feature is low, the resolution of the second image feature obtained after encoding is also low. By fusing the second image feature with other image features, the shallow features included in the obtained pixel-level features are enriched, and the accuracy of the corresponding category of each point in the pixel-level features is improved.

[0084] like Figure 4 As shown in the above Figure 3 Based on the illustrated embodiment, step 2022 may include the following steps:

[0085] Step 401: Obtain a third image feature based on the second image feature and the first image feature.

[0086] In this embodiment, the third image feature is obtained by fusing the second image feature with the first image feature (for example, adding or splicing), and the third image feature obtained by fusion adds features that are not present in the first image feature, thereby enriching the deep features.

[0087] Step 402 : Perform feature fusion processing on the third image feature and other image features of the plurality of image features except the first image feature to obtain pixel-level features.

[0088] Segmentation is the process of predicting which category a pixel in an image belongs to. The features of each pixel are called pixel-level features. In this embodiment, by fusing the second image features and the first image features, the third image features obtained are enriched with deep features, and the third image features are fused with multiple other image features. Since the other image features correspond to different resolutions, multi-resolution features are fused. Therefore, each point in the obtained pixel-level features reflects more features in the image to be processed, thereby improving the accuracy of the predicted category.

[0089] Optionally, step 402 in the above embodiment may further include:

[0090] Step a1: performing scaling processing on the third image feature and at least one other image feature among the multiple other image features to obtain multiple fourth image features of the same size.

[0091] Since the resolution of the first image feature is low, the resolution of the third image feature is also low. By scaling multiple other image features and the third image feature, they are processed to the same size (for example, they are all processed to one-eighth the size of the image to be processed, etc.) to facilitate subsequent processing.

[0092] Step a2: performing pixel-by-pixel superposition processing on the plurality of fourth image features to obtain pixel-level features.

[0093] In this embodiment, multiple fourth image features are subjected to pixel superposition processing. Since the multiple fourth image features have been scaled to the same size, the efficiency of the pixel superposition processing is improved, and pixel-level features are quickly obtained. In addition, by superposing the multiple fourth image features pixel by pixel, multi-resolution features are fused, thereby improving the accuracy of the obtained pixel-level features.

[0094] like Figure 5 As shown in the above Figure 2 Based on the illustrated embodiment, step 204 may include the following steps:

[0095] Step 2041 : Determine a category corresponding to each object in the at least one object based on each object embedding in the at least one object embedding.

[0096] Optionally, each object embedding represents a predicted object, so each object embedding has its own category and center point in the image.

[0097] Step 2042 : Determine a mask corresponding to each object in the at least one object based on the pixel-level features and the at least one object embedding.

[0098] In this embodiment, mask predictions are obtained by interacting object embeddings and pixel-level features (for example, performing matrix correlation operations, etc.). Since each object embedding represents a recognized object, there is no need to predict multiple candidate boxes for each object as in the prior art, which reduces redundant predictions. In the absence of redundant predictions, non-maximum suppression is no longer required, and the network processing speed is faster.

[0099] Optionally, step 2042 in the above embodiment may further include:

[0100] A matrix correlation operation is performed based on the pixel-level features and the at least one object embedding to determine a mask corresponding to each object in the at least one object.

[0101] In this embodiment, the matrix correlation operation is to calculate the similarity between each object embedding and all points in the pixel-level features, so as to extract the pixel points belonging to this object, that is, to obtain the mask corresponding to the object; in an optional example, the similarity can be calculated by element-wise subtraction of the object embedding and the pixel-level features; for example, the matrix corresponding to an object embedding is B*N*C*1*1, and the matrix corresponding to the pixel-level features is B*1*C*H*W. The object embedding is automatically filled with B*N*C*H*W, and then element-wise subtraction is performed to calculate the similarity between the embedding and the pixel-level features. The results of areas with high similarity tend to 1, and areas with low similarity tend to 0. After binarization, mask prediction is formed; this embodiment implements mask prediction through matrix correlation operation, avoids adding a new network structure to the instance segmentation network, makes the network structure simple, and makes image instance segmentation efficient.

[0102] Figure 6 FIG. 1 is a flow chart of an example segmentation network training method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 6 As shown, the following steps are included:

[0103] Step 601: Input a training image into an instance segmentation network to obtain a predicted category and a predicted mask corresponding to at least one predicted object in the training image.

[0104] In this embodiment, the network structure of the instance segmentation network can refer to Figure 1 The instance segmentation network provided is used for understanding. Of course, the instance segmentation network in this embodiment can also be other network models that can achieve end-to-end instance segmentation. This embodiment does not limit the specific network structure.

[0105] Step 602: Determine a first loss based on at least one predicted category and at least one true category corresponding to the training image.

[0106] In this embodiment, at least one true category and at least one true mask corresponding to the at least one true category are known for the training image, and a first loss of the instance segmentation network can be determined based on the difference between the predicted category and the true category.

[0107] Step 603: Determine a second loss based on the at least one predicted mask and the at least one true mask corresponding to the training image.

[0108] Since the instance segmentation network not only predicts the category of at least one object included in the training image, but also predicts a predicted mask corresponding to each object, the second loss of the instance segmentation network can be determined by the difference between the predicted mask and the true mask.

[0109] Step 604 : Supervise the training of the instance segmentation network based on the first loss and the second loss.

[0110] Optionally, the first loss and the second loss may be directly added or weighted together to supervise the training of the instance segmentation network, wherein the weights may be preset according to the application scenario.

[0111] The instance segmentation network training method provided in this embodiment determines a first loss based on the predicted category, determines a second loss based on the predicted mask, and trains the instance segmentation network in combination with the first loss and the second loss, and adjusts the network parameters in the instance segmentation network. Due to the combination of the first loss and the second loss, the trained instance segmentation network can not only achieve accurate category prediction, but also achieve accurate mask prediction, thereby improving the network performance of the instance segmentation network.

[0112] Optionally, step 601 in the above embodiment may further include:

[0113] Step b1: extract features from the training image using the first network branch included in the instance segmentation network to obtain multiple predicted image features.

[0114] The processing process in this embodiment is the same as Figure 2 The only difference in the embodiment shown is that the processed image is a training image. Therefore, the implementation and effect of this step can refer to Figure 2 Step 201 in the illustrated embodiment will not be described in detail here.

[0115] Step b2: Process multiple image features using the encoder included in the instance segmentation network to obtain predicted pixel-level features.

[0116] The implementation and effect of this step can be referred to Figure 2 Step 202 in the illustrated embodiment will not be described in detail here.

[0117] Step b3: Processing a first predicted image feature among the multiple predicted image features using an encoder and a decoder included in the instance segmentation network to obtain at least one predicted embedding.

[0118] The implementation and effect of this step can be referred to Figure 2 Step 203 in the illustrated embodiment will not be described in detail here.

[0119] Step b4: determining a predicted category and a predicted mask corresponding to at least one predicted object in the training image based on the predicted pixel-level features and the at least one predicted embedding.

[0120] The implementation and effect of this step can be referred to Figure 2 Step 204 in the illustrated embodiment will not be described in detail here.

[0121] Optionally, in this embodiment, while predicting the category, the pixel center point corresponding to the object is also predicted, so as to optimize the network's classification learning of the object by using the category branch and help the network learn the positioning feature by using the center point branch.

[0122] The instance segmentation process provided in this embodiment is similar to Figure 2 The difference in the embodiments provided is that the images processed are training images, and Figure 2 The illustrated embodiment processes the image to be processed, and the training image has at least one real category and at least one real mask corresponding to the at least one real category, without the need for redundant artificial priors, such as the complex processes of anchor points (anchor) or allocation of positive and negative samples during instance segmentation training in the prior art, which is more concise and efficient.

[0123] Optionally, the training method provided in the above embodiment may further include:

[0124] A third loss is determined based on the at least one predicted embedding and a predicted image feature of the plurality of predicted image features.

[0125] In this embodiment, the decoder part in the instance segmentation network predicts the category and center point of a specific object, as well as the object embedding. The object embedding and the features of the first network branch setting stage are similarly measured and supervised, and the similarity loss is calculated to obtain a third loss. The parameters of the instance segmentation network are updated in combination with the third loss. The third loss can be an L1 loss. For example, matrix subtraction is performed on the object embedding and a predicted image feature, and then the mean is taken to obtain the loss value of the third loss. The third loss in this embodiment does not require additional human prior knowledge, thereby reducing the cost of network training.

[0126] Optionally, step 601 in the above embodiment may further include:

[0127] Step c1: perform weighted summation of the first loss, the second loss, and the third loss to obtain the network loss.

[0128] In this embodiment, the weight values ​​corresponding to the first loss, the second loss, and the third loss can be pre-set according to the actual application scenario to obtain an instance segmentation network suitable for the current application scenario.

[0129] In step c2, the training of the instance segmentation network is supervised based on the network loss.

[0130] Optionally, a gradient descent algorithm can be used to adjust the network parameters of the instance segmentation network. In this embodiment, due to the combination of the first loss, the second loss and the third loss, the training speed of the instance segmentation network and the network performance of the trained instance segmentation network are improved.

[0131] Any instance segmentation and instance segmentation network training method provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any instance segmentation and instance segmentation network training method provided in the embodiments of the present disclosure can be executed by a processor, such as a processor that executes any instance segmentation and instance segmentation network training method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be repeated below.

[0132] Exemplary devices

[0133] Figure 7 FIG. 1 is a schematic diagram of the structure of an example segmentation device provided by an exemplary embodiment of the present disclosure. Figure 7 As shown, the device provided in this embodiment includes:

[0134] The feature extraction module 71 is used to use the first network branch to perform feature extraction on the image to be processed, and obtain multiple image features output by multiple network layers in the first network branch.

[0135] Among them, there is a one-to-one correspondence between multiple network layers and multiple image features.

[0136] The feature encoding module 72 is used to use an encoder to encode the multiple image features obtained by the feature extraction module 71 to obtain pixel-level features.

[0137] The feature decoding module 73 is configured to process a first image feature among the multiple image features obtained by the feature extraction module 71 using an encoder and a decoder to obtain at least one object embedding included in the image to be processed.

[0138] The instance segmentation module 74 is configured to determine a mask and a category corresponding to at least one object in the image to be processed based on the pixel-level features obtained by the feature encoding module 72 and the at least one object embedding obtained by the feature decoding module 73 .

[0139] The instance segmentation device provided by the above-mentioned embodiment of the present disclosure extracts object embedding, through which the category of at least one object in the image can be directly determined, and the object embedding is used to interact with pixel-level features to obtain a mask of at least one object in the image, thereby directly obtaining instance segmentation results and realizing end-to-end image instance segmentation.

[0140] Figure 8 FIG. 1 is a schematic diagram of the structure of an instance segmentation device provided by another exemplary embodiment of the present disclosure. Figure 8 As shown, in the apparatus provided in this embodiment, the feature encoding module 72 includes:

[0141] The encoding processing unit 721 is configured to use an encoder to encode a first image feature among the multiple image features to obtain a second image feature;

[0142] The feature fusion unit 722 is used to fuse the second image feature with multiple image features to obtain pixel-level features.

[0143] Optionally, the feature fusion unit 722 is specifically used to obtain a third image feature based on the second image feature and the first image feature; and perform feature fusion processing on the third image feature and other image features in the plurality of image features except the first image feature to obtain a pixel-level feature.

[0144] Optionally, when the feature fusion unit 722 performs feature fusion processing on the third image feature and other image features among multiple image features except the first image feature to obtain pixel-level features, it is used to perform scaling processing on the third image feature and at least one other image feature among multiple other image features to obtain multiple fourth image features of the same size; and perform pixel-by-pixel superposition processing on the multiple fourth image features to obtain pixel-level features.

[0145] Optionally, the instance segmentation module 74 includes:

[0146] a category prediction unit 741 for determining a category corresponding to each object in the at least one object based on each object embedding in the at least one object embedding;

[0147] The mask prediction unit 742 is configured to determine a mask corresponding to each object in the at least one object based on the pixel-level features and the at least one object embedding.

[0148] Optionally, the mask prediction unit 742 is specifically configured to perform a matrix correlation operation based on the pixel-level features and the at least one object embedding to determine a mask corresponding to each object in the at least one object.

[0149] Figure 9 Schematic diagram of the structure of an example segmentation network training device provided by an exemplary embodiment of the present disclosure. Figure 9 As shown, the device provided in this embodiment includes:

[0150] The network prediction module 91 is used to input the training image into the instance segmentation network to obtain the predicted category and predicted mask corresponding to at least one predicted object in the training image.

[0151] The first loss determination module 92 is used to determine a first loss based on at least one predicted category obtained by the network prediction module 91 and at least one true category corresponding to the training image.

[0152] The second loss determination module 93 is configured to determine a second loss based on at least one predicted mask obtained by the network prediction module 91 and at least one true mask corresponding to the training image.

[0153] The network training module 94 is used to supervise the training of the instance segmentation network based on the first loss determined by the first loss determination module 92 and the second loss determined by the second loss determination module 93.

[0154] The instance segmentation network training device provided in this embodiment determines a first loss based on the predicted category, determines a second loss based on the predicted mask, and trains the instance segmentation network in combination with the first loss and the second loss, and adjusts the network parameters in the instance segmentation network. Due to the combination of the first loss and the second loss, the trained instance segmentation network can not only achieve accurate category prediction, but also accurate mask prediction, thereby improving the network performance of the instance segmentation network.

[0155] The network prediction module 91 is specifically used to use the first network branch included in the instance segmentation network to extract features of the training image to obtain multiple predicted image features; use the encoder included in the instance segmentation network to process the multiple image features to obtain predicted pixel-level features; use the encoder and decoder included in the instance segmentation network to process the first predicted image feature among the multiple predicted image features to obtain at least one predicted embedding; based on the predicted pixel-level features and the at least one predicted embedding, determine the predicted category and predicted mask corresponding to at least one predicted object in the training image.

[0156] Optionally, the device provided in this embodiment further includes:

[0157] A third loss determination module (not shown in the figure) is used to determine a third loss based on the at least one predicted embedding and one predicted image feature of the multiple predicted image features.

[0158] Optionally, the network training module 94 is specifically configured to perform a weighted summation of the first loss, the second loss, and the third loss to obtain a network loss; and supervise the training of the instance segmentation network based on the network loss.

[0159] Exemplary electronic devices

[0160] Below, reference Figure 10 The electronic device according to the embodiment of the present disclosure is described. The electronic device can be any one or both of the first device 100 and the second device 200, or a standalone device independent of them, and the standalone device can communicate with the first device and the second device to receive the collected input signals from them.

[0161] Figure 10 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0162] like Figure 10 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0163] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0164] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the instance segmentation and instance segmentation network training methods of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0165] In one example, the electronic device 10 may further include an input device 13 and an output device 14 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0166] For example, when the electronic device is the first device 100 or the second device 200, the input device 13 may be the microphone or microphone array described above, for capturing input signals from a sound source. When the electronic device is a standalone device, the input device 13 may be a communication network connector, for receiving collected input signals from the first device 100 and the second device 200.

[0167] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0168] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0169] Of course, to simplify, Figure 10 Only some of the components related to the present disclosure in the electronic device 10 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device 10 may further include any other appropriate components according to specific application scenarios.

[0170] Exemplary computer program products and computer-readable storage media

[0171] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the instance segmentation and instance segmentation network training methods according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0172] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0173] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the instance segmentation and instance segmentation network training methods according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0174] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0175] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0176] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0177] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0178] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0179] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0180] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0181] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An instance segmentation method, comprising: Performing feature extraction on the image to be processed using the first network branch to obtain a plurality of image features output by a plurality of network layers in the first network branch; wherein the plurality of network layers correspond to the plurality of image features in a one-to-one manner; Encoding the plurality of image features using an encoder to obtain pixel-level features; Processing a first image feature of the plurality of image features using the encoder and the decoder to obtain at least one object embedding included in the image to be processed; Determining a mask and a category corresponding to at least one object in the image to be processed based on the pixel-level features and the at least one object embedding; The encoding process of the plurality of image features using an encoder to obtain pixel-level features includes: Using the encoder to encode a first image feature among the multiple image features to obtain a second image feature; obtaining a third image feature based on the second image feature and the first image feature; A feature fusion process is performed on the third image feature and other image features of the multiple image features except the first image feature to obtain the pixel-level feature.

2. The method according to claim 1, wherein The obtaining, based on the pixel-level features and the at least one object embedding, a mask and a category corresponding to at least one object in the image to be processed, comprises: determining, based on each object embedding in the at least one object embedding, a category corresponding to each object in the at least one object; A mask corresponding to each object in the at least one object is determined based on the pixel-level features and the at least one object embedding.

3. A method for training an instance segmentation network, comprising: Inputting a training image into an instance segmentation network to obtain a predicted category and a predicted mask corresponding to at least one predicted object in the training image; determining a first loss based on the at least one predicted category and at least one true category corresponding to the training image; determining a second loss based on the at least one predicted mask and at least one true mask corresponding to the training image; Supervising the training of the instance segmentation network based on the first loss and the second loss; Inputting the training image into the instance segmentation network to obtain a predicted category and a predicted mask corresponding to at least one predicted object in the training image includes: Performing feature extraction on the training image using a first network branch included in the instance segmentation network to obtain a plurality of predicted image features; encoding a first predicted image feature among the plurality of predicted image features using an encoder included in the instance segmentation network to obtain a second predicted image feature; fusing the second predicted image feature with the plurality of predicted image features to obtain a predicted pixel-level feature; Processing a first predicted image feature of the plurality of predicted image features using the encoder and the decoder included in the instance segmentation network to obtain at least one predicted embedding; Determine a predicted category and a predicted mask corresponding to at least one predicted object in the training image based on the predicted pixel-level features and the at least one predicted embedding.

4. An instance segmentation apparatus, comprising: a feature extraction module, configured to perform feature extraction on the image to be processed using the first network branch, and obtain a plurality of image features output by a plurality of network layers in the first network branch; wherein the plurality of network layers correspond to the plurality of image features in a one-to-one manner; A feature encoding module, configured to encode the multiple image features obtained by the feature extraction module using an encoder to obtain pixel-level features; a feature decoding module, configured to process a first image feature among the plurality of image features obtained by the feature extraction module using the encoder and the decoder to obtain at least one object embedding included in the image to be processed; an instance segmentation module, configured to determine a mask and a category corresponding to at least one object in the image to be processed based on the pixel-level features obtained by the feature encoding module and the at least one object embedding obtained by the feature decoding module; The feature encoding module includes: an encoding processing unit, configured to use the encoder to perform encoding processing on a first image feature among the plurality of image features to obtain a second image feature; A feature fusion unit is used to obtain a third image feature based on the second image feature and the first image feature; and perform feature fusion processing on the third image feature and other image features of the multiple image features except the first image feature to obtain the pixel-level feature.

5. An instance segmentation network training device, comprising: a network prediction module, configured to input a training image into an instance segmentation network to obtain a predicted category and a predicted mask corresponding to at least one predicted object in the training image; a first loss determination module, configured to determine a first loss based on at least one predicted category obtained by the network prediction module and at least one true category corresponding to the training image; a second loss determination module, configured to determine a second loss based on at least one predicted mask obtained by the network prediction module and at least one true mask corresponding to the training image; a network training module configured to supervise the training of the instance segmentation network based on a first loss determined by the first loss determination module and a second loss determined by the second loss determination module; The network prediction module is specifically used to use the first network branch included in the instance segmentation network to extract features of the training image to obtain multiple predicted image features; use the encoder included in the instance segmentation network to encode the first predicted image feature among the multiple predicted image features to obtain a second predicted image feature; fuse the second predicted image feature with the multiple predicted image features to obtain a predicted pixel-level feature; use the encoder and decoder included in the instance segmentation network to process the first predicted image feature among the multiple predicted image features to obtain at least one predicted embedding; and determine the predicted category and predicted mask corresponding to at least one predicted object in the training image based on the predicted pixel-level feature and the at least one predicted embedding.

6. A computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 3.

7. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 3.

Images