A method and apparatus for instance segmentation

Abstract

The present specification discloses a method and device for instance segmentation. After determining an initial instance mask, initial image features and initial instance features are determined based on the initial instance mask and an image to be segmented. The accuracy of the initial instance features is determined based on the initial image features, the initial instance features, historical image features and historical instance mask features. The initial instance mask is corrected based on the accuracy. The accumulation of errors between the determined instance and the real instance is avoided, and the accuracy of instance segmentation is improved.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to a method and apparatus for instance segmentation. Background Technology

[0002] Currently, with the development of image processing technology, instance segmentation technology has become a key research area due to its assistance in autonomous vehicle control and high-precision map creation in the field of autonomous driving, as well as its role in object segmentation in the field of image processing.

[0003] In existing technologies, one method for detecting and segmenting instances in an image is based on an image sequence. Specifically, the image to be segmented and its historical images in the image sequence are first determined. Then, based on the image features of the image to be segmented, the image features of each historical image, and the instance features of each historical image, the instance features of the image to be segmented are determined. Finally, an instance mask for the image to be segmented is determined based on the determined instance features. Summary of the Invention

[0004] This specification provides an instance segmentation method and apparatus to partially solve the aforementioned problems existing in the prior art.

[0005] The following technical solution is adopted in this specification:

[0006] This specification provides an instance segmentation method, including:

[0007] The initial instance mask of the image to be segmented is determined based on the continuously acquired images, several historical images preceding the image to be segmented, and historical instance masks in the several historical images.

[0008] Based on the initial instance mask and the image to be segmented, initial image features and initial instance features are determined, wherein the initial instance features are used to characterize the instance classification result of each pixel in the image to be segmented;

[0009] Based on the initial image features, the initial instance features, the features of each historical image, and the features of each historical instance mask, the accuracy of the initial instance features is determined. The accuracy is used to characterize the probability that each pixel in the image to be segmented is the instance classification result corresponding to that pixel.

[0010] The initial instance mask is corrected based on the accuracy.

[0011] Optionally, the accuracy of the initial instance features is determined based on the initial image features, the initial instance features, the features of each historical image, and the features of each historical instance mask, specifically including:

[0012] Based on the features of the initial image and the features of each historical image, the similarity between the initial image and each historical image is determined.

[0013] Based on the similarity scores and the features of each historical instance mask, the predicted instance features of each historical image for the image to be segmented are obtained.

[0014] The accuracy of the initial instance features is determined based on the predicted instance features and the initial instance features.

[0015] Optionally, the accuracy of the initial instance features is determined based on the predicted instance features and the initial instance features, specifically including:

[0016] The initial instance mask is encoded to determine the features of the reference mask;

[0017] The accuracy of the initial instance features is determined based on the predicted instance features, the initial instance features, and the reference mask features.

[0018] Optionally, based on the aforementioned similarities and the features of each historical instance mask, the predicted instance features of each historical image for the image to be segmented are obtained, specifically including:

[0019] Based on historical images, determine the image sequence;

[0020] Based on the features of each historical image and the features of each historical instance mask, determine the image feature sequence and instance feature sequence corresponding to the image sequence;

[0021] Based on the initial image features and the image feature sequence, a similarity sequence is determined;

[0022] Based on the similarity sequence and the instance feature sequence, the predicted instance features are obtained.

[0023] Optionally, the method further includes:

[0024] The image to be segmented and the corrected initial instance mask are stored as historical images and historical instance masks in the historical reference set;

[0025] From the continuously acquired images, the images located after the image to be segmented are re-determined, and instance segmentation is performed on the re-determined image to be segmented based on the historical reference set.

[0026] Optionally, based on the initial instance mask and the image to be segmented, initial image features are determined, specifically including:

[0027] The initial instance and the image to be segmented are used as inputs to a pre-trained encoder to obtain the initial image features output by the encoder.

[0028] The temporal characteristics of the image to be segmented are determined based on the number of frames in the continuously acquired images.

[0029] The positional features of each pixel in the image to be segmented are determined, and the initial image features are updated based on the temporal features and the positional features.

[0030] Optionally, the initial instance mask may be corrected based on the accuracy, specifically including:

[0031] Based on the accuracy, for each pixel in the image to be segmented, it is determined whether the accuracy of that pixel is greater than a preset probability threshold.

[0032] If so, maintain the instance classification of the pixel in the initial instance mask;

[0033] If not, delete the instance category of the pixel in the initial instance mask.

[0034] Optionally, the segmentation model can be trained in the following manner, wherein:

[0035] Training samples are determined from historically continuously acquired images, and the annotations of the training samples are determined. The training samples include a first image, a plurality of second images preceding the first image, and a second instance mask in the plurality of second images. The second images are historical images of the first image.

[0036] The training samples are input into the instance determination layer of the segmentation model to be trained to obtain the first instance mask determined based on the first image, the several historical images and each historical instance mask.

[0037] The first instance mask and the first image are input into the encoding layer of the segmentation model to obtain the first image features and the first instance features of the training samples;

[0038] The first image features, the first initial instance features, the features of each second image, and the features of each second instance mask are input into the correction layer of the segmentation model to obtain the accuracy of the first instance features. Based on the accuracy, the first instance mask is corrected to obtain the target instance mask.

[0039] The segmentation model is trained based on the target instance mask and annotation of the training samples; the segmentation model is used to correct the initial instance mask of the image to be segmented.

[0040] This specification provides an instance segmentation device, specifically including:

[0041] The instance determination module is used to determine the initial instance mask of the image to be segmented based on the image to be segmented in continuously acquired images, several historical images before the image to be segmented, and historical instance masks in the several historical images.

[0042] The feature extraction module is used to determine initial image features and initial instance features based on the initial instance mask and the image to be segmented. The initial instance features are used to characterize the instance classification result of each pixel in the image to be segmented.

[0043] The verification module is used to determine the accuracy of the initial instance features based on the initial image features, the initial instance mask, the features of each historical image, and the features of each historical instance mask. The accuracy is used to characterize the probability that each pixel in the image to be segmented is the instance classification result corresponding to that pixel.

[0044] A correction module is used to correct the initial instance mask based on the accuracy.

[0045] The computer-readable storage medium provided in this specification stores a computer program that, when executed by a processor, implements any of the above-described instance segmentation methods.

[0046] The electronic device provided in this specification includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described instance segmentation methods.

[0047] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0048] In the instance segmentation method provided in this specification, after determining the initial instance mask, the initial image features and initial instance features are determined based on the initial instance mask and the image to be segmented. Then, the accuracy of the initial instance features is determined based on the initial image features, the initial instance features, the features of historical images, and the features of historical instance masks, so as to correct the initial instance mask based on the accuracy.

[0049] As can be seen from the above method, this method avoids the accumulation of errors between the identified instances and the real instances, and improves the accuracy of instance segmentation. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0051] Figure 1 This document provides a flowchart illustrating the method for instance segmentation.

[0052] Figure 2 A schematic diagram of the example mask provided in this specification;

[0053] Figure 3 This is a schematic diagram illustrating the determination of predictive instance features provided in this specification;

[0054] Figure 4 This is a schematic diagram of the structure of the instance segmentation model provided in this specification;

[0055] Figure 5 A schematic diagram of the device for segmenting examples provided in this specification;

[0056] Figure 6 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0058] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0059] Currently, instance segmentation technology is applied in multiple fields, especially in video intelligent analysis and autonomous driving. Instance segmentation methods can be used to help separate target objects from the background in videos, help identify obstacles around autonomous vehicles, determine the movement strategy of autonomous vehicles based on the identified obstacles, and also be used to identify each instance in an image and the correspondence between each instance when creating high-precision maps. Combined with semantic models, it can identify objects in the image that can be used to build maps (e.g., fixed objects such as buildings and traffic signs).

[0060] Generally, instance segmentation methods can be applied to continuously acquired images, performing instance segmentation on each frame of the continuously acquired images to achieve target tracking, etc. Currently, a common instance segmentation method is based on the first image in the continuously acquired images and its instance mask.

[0061] Specifically, one can first acquire continuously collected images and determine the image sequence corresponding to the continuously collected images.

[0062] Then, the first image in the image sequence and the instance mask of that first image are determined.

[0063] Finally, for each image in the image sequence, the instance mask of the image is determined based on the similarity between the image and the historical images preceding it, as well as the historical instance masks corresponding to each historical image.

[0064] However, since the instance mask of each image in the image sequence, except for the first image, is determined based on the historical images of that image and the historical instance masks corresponding to each historical image, the gap between the determined instance mask and the real instance mask will gradually increase as the number of image frames in the image sequence gradually increases, resulting in a gradual decrease in segmentation accuracy.

[0065] The instance mask of the first image can be determined manually, determined by other instance segmentation algorithms, or determined by semantic segmentation algorithms. Determining the instance mask of a single image based on a single image is a relatively mature technology, and this specification will not elaborate further.

[0066] Unlike current methods that rely solely on the similarity between the image to be segmented and its historical images from continuously acquired images, as well as the historical instance masks of each historical image, to determine the instance mask of the image to be segmented, this method addresses the issue where the difference between the determined instance masks and the true mask gradually increases with the number of image frames, leading to a gradual decrease in segmentation accuracy. This new instance segmentation method uses the instance mask determined by the aforementioned method as the initial instance mask, and then refines this initial instance mask based on the historical images of the image to be segmented, thereby reducing the accumulation of errors.

[0067] Figure 1 This is a flowchart illustrating the instance segmentation method provided in this specification, which specifically includes the following steps:

[0068] S100: Determine the initial instance mask of the image to be segmented based on the continuously acquired images, several historical images before the image to be segmented, and historical instance masks in the several historical images.

[0069] In one or more embodiments provided in this specification, the instance segmentation technique can be applied to determine instance masks in continuously acquired images, thereby achieving instance segmentation of the continuously acquired images. Similarly, the instance segmentation technique provided in this specification can be applied to determine instance masks in continuously acquired images.

[0070] Instance segmentation technology is commonly used in the field of autonomous driving, and therefore can generally be executed by the autonomous vehicle or a backend server communicating with the autonomous vehicle. Similarly, the instance segmentation technology provided in this specification can also be executed by the autonomous vehicle or a backend server. For ease of description, this specification will use the server executing the instance segmentation method as an example for subsequent explanations.

[0071] In one or more embodiments provided in this specification, after acquiring a current image, the autonomous vehicle can send the current image to a server. The server can then receive the current image sent by the autonomous vehicle and use it as the image to be segmented. Simultaneously, the server can use images received from the autonomous vehicle at historical times as corresponding historical images to be segmented.

[0072] Specifically, the server can use the images continuously collected by the data acquisition devices on the unmanned vehicle as continuously acquired images.

[0073] Secondly, the server can determine the image that is closest to the current time from the continuously acquired images as the image to be segmented.

[0074] Then, the server can use the images in the continuously acquired images that are before the segmentation to be performed as the historical images of the image.

[0075] Finally, the server can determine the initial instance mask of the image to be segmented based on the image to be segmented, several historical images, and historical instance masks in the several historical images.

[0076] The historical instance mask is the same size as the historical image, such as... Figure 2 As shown.

[0077] Figure 2 This is a schematic diagram of the instance mask provided in this specification. In the diagram, the left side is the image, and the right side is a mask instance of the image. Clearly, the instance mask size is the same as the image size. Of course, the diagram only illustrates the example of an instance mask containing one type of instance. When the image contains multiple instances, different pixels can be used to represent different types of instances, or a specific instance mask can be determined for each instance. The specific number of instances included in the instance mask, the types of instances, and how they are represented can be set as needed; this specification does not impose any restrictions on this.

[0078] Of course, in determining the image to be segmented, the above-mentioned method can also involve acquiring images from the data acquisition device on the autonomous vehicle, storing each image according to the acquisition time, and sending the acquired images to the server for instance segmentation after the autonomous vehicle completes its delivery task. The server stores the received images as continuously acquired images, and when instance segmentation is needed, it determines the image to be segmented and its historical images from the stored continuously acquired images. Based on the image to be segmented, several historical images of the image to be segmented, and historical instance masks of several historical images, it determines the initial instance mask of the image to be segmented.

[0079] The specific methods for determining continuously acquired images, as well as for identifying the image to be segmented and its historical images from the continuously acquired images, can be set as needed; this manual does not impose any restrictions on this.

[0080] In one or more embodiments provided in this specification, as described above, the instance segmentation method in this specification modifies the instance mask of the image to be segmented based on the instance masks of historical images after determining the instance mask of the image to be segmented. Based on this, after determining the image to be segmented and its historical images, the server can determine the initial mask instance of the image to be segmented.

[0081] Specifically, the server can determine specified image features and specified instance features of the image to be segmented. The specified image features include texture, color, and other content in the image to be segmented. The specified instance features contain information used to determine the instance segmentation result for each pixel in the image to be segmented.

[0082] Secondly, for each of the aforementioned identified historical images, the server can determine the features of the historical image and the features of the historical instance mask based on the historical image and its corresponding historical instance mask. The features of the historical image include texture, color, and other content within the image. The features of the historical instance mask are used to characterize the instance segmentation result for each pixel in the image to be segmented.

[0083] Then, the server can obtain the auxiliary features of each historical image for the specified image based on the similarity between the features of each historical image and the features of the specified image, as well as the features of each historical instance mask, and update the features of the specified instance based on the auxiliary features.

[0084] Finally, the server can determine the initial instance mask for the image to be segmented based on the updated specified instance features.

[0085] Additionally, it should be noted that in this specification, the image to be segmented and several historical images of that image can be sequentially continuous. Therefore, in scenarios where the motion strategy of an autonomous vehicle is determined, the historical images of the image to be segmented can be determined by selecting a specified number of most recently acquired images, based on the acquisition time of each frame, in an order from nearest to farthest.

[0086] Furthermore, since instance segmentation techniques typically require historical instance masks from historical images to determine the initial instance mask, the server can also determine the historical instance masks in each historical image while determining the historical images. These instance masks can be manually pre-annotated or obtained through instance segmentation algorithms.

[0087] Of course, the continuously acquired images can be transmitted from the unmanned vehicle to the server in real time after the acquisition device captures the images, or the server can send an acquisition request to the unmanned vehicle when instance segmentation is required, so that the unmanned vehicle can determine a specified number of frames from the continuously acquired images and return them to the server. The specific way the server acquires images can be configured as needed, and this manual does not impose any restrictions.

[0088] In this specification, "unmanned vehicle" can refer to unmanned vehicles, robots, automated delivery equipment, or other devices capable of autonomous driving. Based on this, unmanned driving equipment using the target recognition method provided in this specification can be used to perform delivery tasks in the delivery field, such as in business scenarios involving express delivery, logistics, and food delivery. To ensure the safe operation of unmanned driving equipment in delivery scenarios, it is necessary to identify targets such as traffic lights, traffic signs, and other vehicles in the surrounding environment using the acquired images.

[0089] S102: Based on the initial instance mask and the image to be segmented, determine the initial image features and the initial instance features, wherein the initial instance features are used to characterize the instance classification result of each pixel in the image to be segmented.

[0090] In one or more embodiments provided in this specification, the initial instance mask is actually an image that can characterize an instance; therefore, modifying the initial instance mask is equivalent to modifying the image. Image features, being the result of feature extraction from the image by the server, are obviously easier for the server to learn how to modify based on the extracted features, rather than modifying the image itself. Therefore, the server can determine the features used to characterize the instance segmentation result for each pixel in the image to be segmented, and determine the accuracy of these features to modify the initial instance mask. The features of each historical image and the features of each historical instance mask are determined for each historical image based on that historical image and its corresponding historical instance mask. Based on this, the server can use the same method to determine the initial image features and initial instance features of the image to be segmented.

[0091] Specifically, the server first obtains the image to be segmented, as well as the initial instance mask determined in step S100 above.

[0092] Then, the server can take the image to be segmented and the initial instance mask as input, input them into a pre-trained feature extraction model, and obtain the initial image features and initial instance features output by the feature extraction model.

[0093] The feature extraction model can be a neural network model, a recurrent neural network model, etc. The specific model structure can be set as needed, and this manual does not impose any restrictions on it.

[0094] Of course, the methods used to determine the initial image features and initial instance features mentioned above are the same methods used to determine the features of historical images and the features of historical instance masks mentioned above.

[0095] In addition, since the encoder structure can be used to feature / vectorize content such as images and text based on a specific encoding structure, the server can use the encoder structure to implement the above steps of determining the initial image features and the initial instance features.

[0096] Specifically, the server can use the image to be segmented and the initial instance mask as input to a pre-trained encoder to obtain the initial image features and initial instance features output by the encoder.

[0097] It should be noted that if an encoder structure is used to determine image features and instance features, the input of the encoder that determines the first instance features and the first image features is only the image to be segmented, while the input of the encoder that determines the initial instance features, the initial image features, the features of historical images, and the features of historical instance masks is the image and the mask instance corresponding to the image. That is to say, compared with the first image features determined based on the image to be segmented in step S100, the initial image features determined in step S102 include not only features such as color and texture, but also relevant features of the mask instance.

[0098] S104: Based on the initial image features, the initial instance features, the features of each historical image, and the features of each historical instance mask, determine the accuracy of the initial instance mask. The accuracy is used to characterize the probability that each pixel in the image to be segmented is the instance classification result corresponding to that pixel.

[0099] In one or more embodiments provided in this specification, generally, the closer two images are, the smaller the difference between the instance masks corresponding to these two images should be; conversely, the lower the similarity between two images, the larger the difference between the instance masks corresponding to these two images should be. If the similarity between two images is low, but the similarity between their corresponding instance masks is high, the determined instance mask may not be accurate enough.

[0100] Based on this, the server can determine the accuracy of the initial instance mask based on the similarity between the initial image features of the image to be segmented and the features of each historical image, as well as the similarity between the initial instance features and the features of each historical instance mask.

[0101] Specifically, for each historical image, the server can determine the similarity between the historical image and the image to be segmented based on the features of the historical image and the features of the initial image corresponding to the image to be segmented.

[0102] Then, the server can determine the accuracy between the historical instance mask and the initial instance mask based on the features of the historical instance mask corresponding to the historical image and the features of the initial instance.

[0103] Finally, the server can compare the similarity between the historical image and the image to be segmented, as well as the similarity between the historical instance mask and the initial instance mask, and take the parts that are the same or similar as the accurate parts, and the parts that are significantly different as the inaccurate parts.

[0104] The aforementioned similarity can be obtained by normalizing the result of feature multiplication. In other words, the obtained similarity is a matrix that can be used to characterize the similarity between each pixel in the image to be segmented and the historical images.

[0105] Furthermore, since the motion of objects is usually continuous, for each frame of a continuously acquired image, instances appearing in that frame have a high probability of appearing in historical images within that frame. For each historical image, the higher the similarity between the historical image and the current frame, the higher the probability that the historical instance mask of the historical image is the instance mask of the current frame, the higher the probability that the same instance appears in both the historical image and the current frame, and the more useful the historical image becomes.

[0106] Therefore, the server can query the most relevant and useful information about the current frame from the historical images based on the image features of the image to be segmented and the image features of each historical image. It then integrates the determined information to obtain a prediction of the image to be segmented based on the features of each historical image. The server can then verify the initial instance features of the image to be segmented based on this prediction.

[0107] Based on this, the server can obtain the predicted instance features of the image to be segmented by considering the similarity between the initial image features and the features of each historical image, as well as the features of each historical instance mask.

[0108] Specifically, for each historical image determined in step S100, the server can determine the similarity between the features of the target image to be segmented and the features of the historical image. This similarity can be a feature product, cosine distance, Euclidean distance, etc., and the specific method for determining this similarity can be set as needed; this specification does not impose any restrictions on it.

[0109] The server can then determine the weight of the historical image corresponding to the image to be segmented based on the similarity, where the similarity and weight are positively correlated.

[0110] Finally, the server can determine the predicted instance features based on the weights corresponding to each historical image and the features of the historical instance masks corresponding to each historical image.

[0111] The server can perform a weighted summation based on the weights corresponding to each historical image and the features of each historical instance mask, and use the weighted result as the predicted instance features.

[0112] In addition, after determining the predicted instance features, the server can determine the accuracy of the initial instance features based on the predicted instance features.

[0113] Specifically, the server can determine the similarity between the feature values ​​of the predicted instance features and the initial instance features, normalize the similarity, and use the normalized result as the similarity of the initial instance features.

[0114] Of course, the server can also use the predicted instance features and the initial instance features as input to a pre-trained accuracy determination model to obtain the accuracy of the initial instance features output by the accuracy determination model. The accuracy is used to characterize the probability that each pixel in the image to be segmented is the instance classification result corresponding to that pixel.

[0115] S106: Based on the accuracy, the instance mask is corrected.

[0116] In one or more embodiments provided in this specification, after determining the accuracy, the server can modify the instance mask based on the accuracy.

[0117] Specifically, the server determines whether the accuracy of each pixel in the image to be segmented is greater than a preset probability threshold.

[0118] If so, maintain the pixel's position within the instance classification of the initial instance mask.

[0119] If not, remove the instance category of the pixel's location in the initial instance mask.

[0120] Taking an image to be segmented containing pixels A and B as an example, assuming a probability threshold of 80%, if the probability corresponding to the location of pixel A is 90% and the probability corresponding to the location of pixel B is 10% in terms of accuracy, then the server can retain the instance classification of pixel A in the initial instance mask and delete the instance classification of pixel B in the initial instance mask. Deleting the instance classification of pixel B in the initial instance mask means adjusting the instance classification corresponding to the location of pixel B to background.

[0121] Of course, in addition to the above-mentioned correction methods, the server can also use methods such as adjusting the position of instances in the diagram to correct the initial instance mask.

[0122] Furthermore, after correcting the initial instance mask, the server can store the image to be segmented and the corrected initial instance mask as historical images and their corresponding historical instance masks in a historical reference set. This historical reference set stores each historical image and its corresponding historical instance mask, and / or the features of each historical image and its corresponding historical instance mask.

[0123] The server can then redetermine the image to be segmented from the images following the image to be segmented in the continuously acquired images. Based on the redetermined image and the historical reference set, it can determine the initial instance mask of the image to be segmented and correct the initial instance mask. That is, instance segmentation is performed on the redetermined image to be segmented based on the historical reference set.

[0124] based on Figure 1 The instance segmentation method determines initial instance features and initial instance features based on the initial instance mask and the image to be segmented. Then, it corrects the initial instance mask based on the accuracy of the initial instance features determined by the initial image features, initial instance features, historical image features, and historical instance mask features. This avoids the accumulation of errors between the determined instances and the real instances and improves the accuracy of instance segmentation.

[0125] Furthermore, in this specification, the weighted calculation of predicted instance features based on the weights corresponding to each historical image and the features of each historical instance mask will inevitably cause some information loss. Therefore, in step S104, the server can also use different channels to represent different historical images based on feature dimensions, and then determine the predicted instance features based on the features of each channel.

[0126] Specifically, the server can determine the image sequence based on each historical image.

[0127] Then, the server can determine the image feature sequence and instance feature sequence corresponding to the image sequence based on the features of each historical image and the features of each historical instance mask.

[0128] Finally, the server can determine a similarity sequence based on the initial image features and the image feature sequence, and determine the predicted instance features based on the similarity sequence and the instance feature sequence.

[0129] Assume R C×W×H The feature map representing the image frame is given by R. If the number of historical images is T, then the feature map sequence corresponding to the image sequence determined by the historical images corresponding to the image to be segmented can be represented by R. T×C×W×H To characterize.

[0130] Therefore, assuming the number of historical images is T, K represents image features, and V represents instance features, the initial image features of the image to be segmented can be characterized as follows: The initial instance features can be characterized as In determining the initial image features, 1 / 8 downsampling was used, while in determining the initial image features, 1 / 2 downsampling was used. Therefore, the image feature sequence corresponding to the image sequence is: Instance feature sequence is Where C represents the number of channels in the feature map, T represents the number of historical images in the image sequence, and W and H represent the length and width dimensions of the feature map.

[0131] Therefore, the server can determine a similarity sequence (or similarity matrix) based on the initial image features and the image feature sequence, and then determine the predicted instance features based on the similarity sequence and the instance feature sequence. For example... Figure 3 As shown.

[0132] In the figure, the image feature sequence and the initial image features determine the similarity sequence, and the similarity sequence and the instance feature sequence determine the predicted instance features. The dimensions and sizes of the predicted instance features are the same as those of the initial instance features.

[0133] Furthermore, after determining the image feature sequence, since the features contained in the feature map mainly represent the features of image pixels, and the position of the instance in the image is also an important factor affecting the determination of the tracking result when performing instance segmentation, in step S104, in order to improve the accuracy and efficiency of instance segmentation, the server can also determine the position features of each pixel in each image in the image sequence to obtain the spatiotemporal feature sequence corresponding to the image sequence.

[0134] Specifically, in this specification, the encoder used to determine image features can be a transformer encoder, that is, an encoder in the encoder-decoder structure of a transformer model within a Recurrent Neural Network (RNN). This encoder has a positional encoding function, allowing the server to encode each pixel in each image of the image sequence based on its position within the image and the order of the images within the image sequence. This encoding yields a processing result containing both positional and temporal features. Based on the processing results for each pixel, the server determines the spatiotemporal feature sequence corresponding to the image sequence. Furthermore, since the determined spatiotemporal feature sequence is used to supplement the missing positional and temporal features in the image feature sequence determined in step S104, its dimension is the same as the dimension of the image feature sequence. In other words, it can supplement the processing result containing both positional and temporal features for each feature in the image feature sequence.

[0135] Furthermore, in this specification, since the encoder encodes based on the similarity between features in the input image sequence, the features of the instance can be more significant after the decoder decodes, thereby making the final instance mask more accurate. Therefore, in order to facilitate the encoder to encode based on similarity, after the server obtains the image feature sequence in step S104, it can also perform dimensionality reduction processing on the image feature sequence using a preset one-dimensional convolution kernel.

[0136] In addition, when determining the spatiotemporal feature sequence, the server can determine the positional feature for each pixel by using cosine and sine processing with different frequencies for the pixel's x-axis and y-axis coordinates in the image.

[0137] Specifically, it can be done through the formula. Determine the location features, where PE pos Indicates location features, `pos` represents the position of the pixel in the corresponding dimension (x-axis or y-axis), and `d` is the number of channels in the feature map sequence. It should be noted that since this specification supplements information from both temporal and spatial perspectives, and the spatial aspect includes both x-axis and y-axis dimensions, when supplementing spatiotemporal information, it needs to be described from three dimensions: x-axis, y-axis, and z-axis (time dimension). Therefore, the aforementioned number of channels needs to be divisible by 3. If the image sequence is supplemented from other dimensions, then `d` should be divisible by those other quantities to ensure that the dimensions of the resulting spatiotemporal feature sequence are consistent with the dimensions of the image feature sequence.

[0138] In one or more embodiments provided in this specification, after determining the spatiotemporal feature sequence and the image feature sequence, the server can correspondingly expand the spatiotemporal feature sequence into the image feature sequence, so that the features in the image feature sequence carry both pixel features and the position of the pixel in the image and the temporal sequence of the image where the pixel is located. Then, the expanded image feature sequence can be used as input to the encoder to obtain the encoding result.

[0139] Specifically, firstly, the server can add each spatiotemporal feature contained in the spatiotemporal feature sequence to each image feature contained in the image feature sequence with the same dimension, according to the dimension of the spatiotemporal feature sequence. In other words, it expands the corresponding spatiotemporal features of each image feature contained in the image feature sequence.

[0140] It should be noted that the aforementioned time dimension can be the number of frames for each image in a series of acquired images, or it can be based on the number of frames for each image in an image sequence. Furthermore, the aforementioned method of supplementing spatiotemporal features can also be used to supplement the initial image features and the first image features.

[0141] In addition, the above instance segmentation method can also be implemented using an instance segmentation model.

[0142] That is, the image to be segmented, historical images of the image to be segmented, and the masks corresponding to each historical image are taken as inputs and input into the instance segmentation model to obtain the instance mask of the image to be segmented output by the instance segmentation model.

[0143] The server can then train the instance segmentation model in the following manner:

[0144] Specifically, the server can determine training samples from historically continuously acquired images and determine the annotations of the training samples. The training samples include a first image, several second images preceding the first image, and second instance masks in the several second images.

[0145] Secondly, the server can input the training sample into the instance determination layer of the segmentation model to be trained, obtain the first instance mask determined based on the first image, several second images and each second instance mask, and input the first instance mask and the first image into the encoding layer of the segmentation model to obtain the first image features and the first instance features of the training sample.

[0146] Then, the server can input the first image features, the first initial instance features, the features of each second image, and the features of each second instance mask into the correction layer of the segmentation model to obtain the accuracy of the first instance features, and correct the first instance mask according to the accuracy to obtain the target instance mask.

[0147] Finally, the segmentation model is trained based on the target instance mask and annotation of each training sample, so as to complete the subsequent step of correcting the initial instance mask of the image to be segmented based on the trained segmentation model.

[0148] Of course, the above-mentioned steps for determining accuracy can also be implemented by the server inputting the features of the first image, the features of each first image, and the features of each historical mask into the auxiliary prediction layer of the segmentation model to obtain the predicted instance features of the training sample. Then, the server inputs the predicted instance features and the initial instance features into the correction layer of the segmentation model to obtain the accuracy of the initial instance features.

[0149] In addition, to ensure the accuracy of the accuracy determined above, the server can also use the characteristics of the determined initial instance mask to assist in determining the accuracy.

[0150] Specifically, the server can encode the initial instance mask and determine the reference mask features corresponding to the initial instance mask.

[0151] The server can then determine the similarity between the reference mask features, the predicted instance features, and the initial instance features, and determine the accuracy based on the similarity.

[0152] Of course, the reference mask features, predicted instance features, and initial instance features can also be used as inputs to the correction layer of the segmentation model to obtain the accuracy of the initial instance mask output by the correction layer.

[0153] Following the same approach, this specification provides a schematic diagram of an instance segmentation model that includes a memory layer, such as... Figure 4As shown.

[0154] Figure 4 This is a schematic diagram of the instance segmentation model provided in this specification. From continuously acquired images, an image to be segmented is determined. Based on this image, first image features Kt and first instance features Vt are determined. Then, based on the features of each historical image corresponding to each historical image stored in the memory layer, the features of each historical instance mask, and the first image features, the first instance features are updated to obtain fused features. Finally, an initial instance mask is obtained based on these fused features. The features stored in the memory layer are the same as the features of each historical image and each historical instance mask in the aforementioned historical reference set.

[0155] Therefore, based on the initial instance mask and the image to be segmented, the initial image features Kc and initial instance features Vc can be determined. Based on Kc and the features stored in the memory layer, the predicted instance features are determined. The predicted instance features and the target instance features are input into the verification layer to obtain the target instance features output by the verification layer. The target instance mask is determined based on the target instance features, and the target instance features and the initial image features are stored in the memory layer.

[0156] The above describes one or more embodiments of the instance segmentation method provided in this specification. Based on the same idea, this specification also provides an instance segmentation apparatus, including:

[0157] Figure 5 This is a schematic diagram of the instance segmentation device provided in this specification, wherein:

[0158] The instance determination module 200 is used to determine the initial instance mask of the image to be segmented based on the continuously acquired image, several historical images before the image to be segmented, and historical instance masks in the several historical images.

[0159] The feature extraction module 202 is used to determine initial image features and initial instance features based on the initial instance mask and the image to be segmented. The initial instance features are used to characterize the instance classification result of each pixel in the image to be segmented.

[0160] The verification module 204 is used to determine the accuracy of the initial instance features based on the initial image features, the initial instance mask, the features of each historical image, and the features of each historical instance mask. The accuracy is used to characterize the probability that each pixel in the image to be segmented is the instance classification result corresponding to that pixel.

[0161] The correction module 206 is used to correct the initial instance mask according to the accuracy.

[0162] Optionally, the device further includes:

[0163] Training device 208 is used to train a segmentation model in the following manner: training samples are determined from historically continuously acquired images, and the annotations of the training samples are determined. The training samples include a first image, several second images preceding the first image, and second instance masks in the several second images. The second images are historical images of the first image. The training samples are input into the instance determination layer of the segmentation model to be trained to obtain a first instance mask determined based on the first image, the several historical images, and each historical instance mask. The first instance mask and the first image are input into the encoding layer of the segmentation model to obtain a first image feature and a first instance feature of the training samples. The first image feature, the first initial instance feature, the features of each second image, and the features of each second instance mask are input into the correction layer of the segmentation model to obtain the accuracy of the first instance feature. Based on the accuracy, the first instance mask is corrected to obtain a target instance mask. The segmentation model is trained based on the target instance mask and annotations of the training samples. The segmentation model is used to correct the initial instance mask of the image to be segmented.

[0164] Optionally, the verification module 204 is configured to determine the similarity between the initial image and each historical image based on the initial image features and the features of each historical image, obtain the predicted instance features of each historical image for the image to be segmented based on the similarity and the features of each historical instance mask, and determine the accuracy of the initial instance features based on the predicted instance features and the initial instance features.

[0165] Optionally, the verification module 204 is used to encode the initial instance mask, determine the reference mask features, and determine the accuracy of the initial instance features based on the predicted instance features, the initial instance features, and the reference mask features.

[0166] Optionally, the verification module 204 is configured to determine an image sequence based on each historical image, determine an image feature sequence and an instance feature sequence corresponding to the image sequence based on the features of each historical image and the features of each historical instance mask, determine a similarity sequence based on the initial image features and the image feature sequence, and obtain predicted instance features based on the similarity sequence and the instance feature sequence.

[0167] Optionally, the correction module 206 is used to store the image to be segmented and the corrected initial instance mask as historical images and historical instance masks in a historical reference set, redetermine the image to be segmented from the images ranked after the image to be segmented in the continuously acquired images, and perform instance segmentation on the redetermined image to be segmented based on the historical reference set.

[0168] Optionally, the instance determination module 200 is used to take the initial instance and the image to be segmented as input, input a pre-trained encoder to obtain the initial image features output by the encoder, determine the temporal features of the image to be segmented according to the number of frames of the image to be segmented in the continuously acquired images, determine the positional features of each pixel in the image to be segmented, and update the initial image features according to the temporal features and the positional features.

[0169] Optionally, the correction module 206 is used to determine, based on the accuracy, whether the accuracy of each pixel in the image to be segmented is greater than a preset probability threshold. If yes, the pixel's position is kept in the instance classification of the initial instance mask; otherwise, the pixel's position is deleted from the instance classification of the initial instance mask.

[0170] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided instance splitting method.

[0171] This instruction manual also provides Figure 6 The diagram shows a schematic structural representation of the electronic device. Figure 6 At the hardware level, the autonomous driving device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for various operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The instance segmentation method described above. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0172] When this instance segmentation method is applied to autonomous vehicles to assist in the autonomous vehicle control room, the electronic device can also be an autonomous vehicle.

[0173] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0174] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0175] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0176] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0177] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0179] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0180] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0181] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0182] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0183] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0184] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0185] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0187] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0188] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for instance segmentation, characterized in that, Specifically, it includes: The initial instance mask of the image to be segmented is determined based on the continuously acquired images, several historical images preceding the image to be segmented, and historical instance masks in the several historical images. Based on the initial instance mask and the image to be segmented, initial image features and initial instance features are determined, wherein the initial instance features are used to characterize the instance classification result of each pixel in the image to be segmented; Based on the initial image features, the initial instance features, the features of each historical image, and the features of each historical instance mask, the accuracy of the initial instance features is determined. The accuracy is used to characterize the probability that each pixel in the image to be segmented is the instance classification result corresponding to that pixel. Based on the accuracy, the initial instance mask is corrected; Specifically, determining the accuracy of the initial instance features based on the initial image features, the initial instance features, the features of each historical image, and the features of each historical instance mask includes: Based on the features of the initial image and the features of each historical image, the similarity between the initial image and each historical image is determined. Based on the similarity scores and the features of each historical instance mask, the predicted instance features of each historical image for the image to be segmented are obtained. Based on the predicted instance features and the initial instance features, determine the accuracy of the initial instance features; Based on the predicted instance features and the initial instance features, the accuracy of the initial instance features is determined, specifically including: The initial instance mask is encoded to determine the features of the reference mask; The accuracy of the initial instance features is determined based on the predicted instance features, the initial instance features, and the reference mask features.

2. The method as described in claim 1, characterized in that, Based on the aforementioned similarities and the features of each historical instance mask, the predicted instance features of each historical image for the image to be segmented are obtained, specifically including: Based on historical images, determine the image sequence; Based on the features of each historical image and the features of each historical instance mask, determine the image feature sequence and instance feature sequence corresponding to the image sequence; Based on the initial image features and the image feature sequence, a similarity sequence is determined; Based on the similarity sequence and the instance feature sequence, the predicted instance features are obtained.

3. The method as described in claim 1, characterized in that, The method further includes: The image to be segmented and the corrected initial instance mask are stored as historical images and historical instance masks in the historical reference set; From the continuously acquired images, the images located after the image to be segmented are re-determined, and instance segmentation is performed on the re-determined image to be segmented based on the historical reference set.

4. The method as described in claim 1, characterized in that, Based on the initial instance mask and the image to be segmented, the initial image features are determined, specifically including: The initial instance and the image to be segmented are used as inputs to a pre-trained encoder to obtain the initial image features output by the encoder. The temporal characteristics of the image to be segmented are determined based on the number of frames in the continuously acquired images. The positional features of each pixel in the image to be segmented are determined, and the initial image features are updated based on the temporal features and the positional features.

5. The method as described in claim 1, characterized in that, Based on the accuracy, the initial instance mask is corrected, specifically including: Based on the accuracy, for each pixel in the image to be segmented, it is determined whether the accuracy of that pixel is greater than a preset probability threshold. If so, maintain the instance classification of the pixel in the initial instance mask; If not, delete the instance category of the pixel in the initial instance mask.

6. The method according to any one of claims 1 to 5, characterized in that, The segmentation model is trained using the following method, where: Training samples are determined from historically continuously acquired images, and the annotations of the training samples are determined. The training samples include a first image, a plurality of second images preceding the first image, and a second instance mask in the plurality of second images. The second images are historical images of the first image. The training samples are input into the instance determination layer of the segmentation model to be trained to obtain the first instance mask determined based on the first image, the several historical images and each historical instance mask. The first instance mask and the first image are input into the encoding layer of the segmentation model to obtain the first image features and the first instance features of the training samples; The first image features, the first initial instance features, the features of each second image, and the features of each second instance mask are input into the correction layer of the segmentation model to obtain the accuracy of the first instance features. Based on the accuracy, the first instance mask is corrected to obtain the target instance mask. The segmentation model is trained based on the target instance mask and annotation of the training samples; the segmentation model is used to correct the initial instance mask of the image to be segmented.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 6.

Images