Semantic segmentation and interactive annotation method for autonomous driving scenes and related devices

Abstract

The present application discloses a semantic segmentation interactive annotation method and related device for autonomous driving scenes, the method comprising obtaining image embedding and prompt embedding; concatenating the prompt embedding and the initialized prediction token to obtain a token embedding; mapping the token embedding and the image embedding into a transformer module, updating the token embedding through a multi-head attention mechanism and fusing the prompt embedding and the image embedding to generate a context-aware intermediate representation; inputting the updated mask token into the classification head to predict the target type of the region of interest masked by the mask token. The present application does not require a large classification dataset. By simply adding a classification branch, one can fine-tune one's own dataset based on the open source algorithm, effectively solving the technical problems of the prior art of the tandem SAM model and the classification model, such as excessively large classification parameters, slow inference speed, high machine configuration requirements, and impact on the interactive annotation experience.

Description

Technical Field

[0001] The present application belongs to the field of autonomous driving technology, and specifically relates to a semantic segmentation and interactive annotation method for autonomous driving scenes and related devices. Background Art

[0002] The autonomous driving of vehicles relies on the semantic understanding of the scene by the on-board algorithm, so the model requires a large amount of semantic-level annotation information during training. With the rapid development of artificial intelligence, traditional manual annotation methods have gradually evolved into automatic annotation of large models and manual repair of incorrect annotations. The existing automatic annotation algorithms are mainly based on the SAM (SegmentAnything Model) model that was open sourced in 2023. The SAM model can not only implement interactive annotation based on input points, boxes, mask areas, etc. during inference, but also open source a huge dataset SA1B containing 1.1B labels, which enables the SAM model to distinguish almost any structure with clear texture in any picture.

[0003] The SAM model can achieve interactive segmentation, that is, by clicking on any point on the image, the target area can be segmented, only distinguishing the targets without identifying the specific categories of each target. In order to identify the specific category of the target, the SAM model is generally applied to the segmentation and annotation platform in the prior art. The model predicts the approximate range, and then manually labels and repairs the range, or connects complex classification large model algorithms in series to achieve the purpose of segmenting the area and classifying the target. The former has the problem of consuming a lot of manpower and time costs; the latter cannot be retrained according to its own rules because the categories of the serially connected classification models are fixed. At the same time, this method requires the classification model to be generalized to all scenarios. Usually, the number of parameters is huge and dense and large training data is required. Excessive classification parameters will lead to slow inference speed and too high requirements for machine configuration, affecting the experience of interactive annotation. Summary of the Invention

[0004] The purpose of this application is to provide a semantic segmentation interactive annotation method and related devices for autonomous driving scenarios, so as to solve the technical problems in the prior art that the SAM model only distinguishes targets but does not classify them, and requires manual annotation or a cascade classification algorithm. The former is time-consuming and labor-intensive, while the latter has too many training parameters, too slow inference speed, and too high requirements on machine configuration, which affects the experience of interactive annotation.

[0005] To achieve the above objectives, the first aspect of the present application provides a semantic segmentation and interactive annotation method for autonomous driving scenarios, comprising:

[0006] Obtaining an image embedding and a hint embedding, wherein the image embedding includes multi-scale features of the source image and the hint embedding includes an encoding vector of the interaction information;

[0007] Concatenate the hint embedding and the initialized prediction token to obtain a token embedding, wherein the prediction token includes a mask token and an iou token, wherein the mask token is used to predict the mask result and the iou token is used to predict the mask quality;

[0008] The token embedding and the image embedding are mapped and input into the transformer module, the token embedding is updated through a multi-head attention mechanism, and the prompt embedding and the image embedding are fused to generate a context-aware intermediate representation;

[0009] The updated mask token is input into the classification head to predict the target type of the region of interest masked by the mask token.

[0010] In one or more embodiments, further comprising:

[0011] outputting a target mask based on the updated token embedding and the context-aware intermediate representation;

[0012] Based on the target mask and the target type of the region of interest in the mask corresponding to the mask token, the target type of the region of interest in the target mask is output.

[0013] In one or more embodiments, the step of outputting a target mask based on the updated token embedding and the context-aware intermediate representation includes:

[0014] Inputting the updated mask token and the context-aware intermediate representation into a mask prediction head to generate a plurality of initial masks;

[0015] Based on the updated iou token, predict the intersection-over-union ratio of the initial mask and the true mask;

[0016] Embedding the updated token into an input object score module to predict the confidence of the initial mask;

[0017] Based on the initial mask, the intersection-over-union ratio, and the confidence level, a plurality of the initial masks are sorted, and a target mask is output.

[0018] In one or more embodiments, in the step of inputting the updated mask token and the context-aware intermediate representation into a mask prediction head to generate a plurality of initial masks, the input of the mask prediction head further includes a high-resolution feature map.

[0019] In one or more embodiments, the step of outputting a target mask based on the updated token embedding and the context-aware intermediate representation further comprises:

[0020] Embed the updated token into the input object pointer projection layer to obtain an object pointer, where the object pointer is used to identify the location index of the target instance currently being processed in the memory bank;

[0021] The historical mask features stored in the memory bank are obtained based on the object pointer index, and a cross-attention calculation is performed with the target mask to correct the target mask.

[0022] In one or more embodiments, in the step of updating the token embedding and fusing the prompt embedding and the image embedding to generate a context-aware intermediate representation through a multi-head attention mechanism,

[0023] The transformer module is a two-layer encoder, each layer of the encoder sequentially performs self-attention processing on the token embedding, performs cross-attention processing from the token embedding to the image embedding, generates new features through a multi-layer perceptron, and performs cross-attention processing from the image embedding to the token embedding;

[0024] In each attention process, the image embedding is superimposed with the position encoding once, and the token embedding is superimposed with the initial token embedding once.

[0025] In one or more embodiments, the classification head is a lightweight multi-layer perceptron including one hidden layer.

[0026] To achieve the above objectives, the second aspect of the present application provides a semantic segmentation and interactive annotation device for autonomous driving scenarios, comprising:

[0027] an acquisition module for acquiring an image embedding and a hint embedding, wherein the image embedding includes multi-scale features of the source image and the hint embedding includes an encoding vector of the interaction information;

[0028] a splicing module for splicing the hint embedding and the initialized prediction token to obtain a token embedding, wherein the prediction token includes a mask token and an iou token, wherein the mask token is used to predict the mask result and the iou token is used to predict the mask quality;

[0029] A decoding module is configured to input the mapping of the token embedding and the image embedding into a transformer module, update the token embedding through a multi-head attention mechanism, and fuse the prompt embedding and the image embedding to generate a context-aware intermediate representation;

[0030] The mask classification module is used to input the updated mask token into the classification head and predict the target type of the region of interest masked by the mask token.

[0031] In order to achieve the above-mentioned objectives, the third aspect of the present application provides an electronic device, including:

[0032] at least one processor; and

[0033] A memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to execute the semantic segmentation interactive annotation method as described in any of the above embodiments.

[0034] In order to achieve the above-mentioned objectives, the fourth aspect of the present application provides a machine-readable storage medium, which stores executable instructions. When the instructions are executed, the machine executes the semantic segmentation interactive annotation method as described in any of the above-mentioned embodiments.

[0035] Different from the prior art, the present invention has the following advantages:

[0036] This application implements the functions of semantic segmentation and classification simultaneously by inputting the updated mask token into the classification header. It does not require a huge classification dataset. By simply adding classification branches, one can fine-tune one's own dataset based on the open source algorithm. This effectively solves the technical problems in the existing technology of the tandem SAM model and classification model, such as excessively large classification parameters, slow inference speed, high machine configuration requirements, and impact on the interactive labeling experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a flowchart of an implementation method of the semantic segmentation interactive annotation method for autonomous driving scenarios of the present application;

[0039] Figure 2 yes Figure 1 A schematic flow chart of an implementation method corresponding to S500;

[0040] Figure 3 This is a structural diagram of an implementation scheme of the semantic segmentation interactive annotation model of the present application;

[0041] Figure 4 This is a segmentation result diagram of an embodiment of the semantic segmentation interactive annotation method of the present application;

[0042] Figure 5 This is a structural diagram of an embodiment of a semantic segmentation and interactive annotation device for autonomous driving scenarios of the present application;

[0043] Figure 6 It is a structural diagram of an embodiment of the electronic device of the present application. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0045] Because the SAM model only distinguishes targets and does not identify the specific categories of each target, for example, it can simultaneously segment four people and four cars in an image, resulting in eight segmentation results, but these eight segmentation results are category-agnostic. Currently, automatic labeling algorithms based on the SAM model obtain the target category and region by adding a classification model and concatenating the results of the two models. For example, a detection model DINO is added before the SAM model to detect any object in the image, obtain the object's category information and bounding box, and then input the bounding box as interactive information into the SAM model to achieve image instance segmentation. There are also models such as SAM-CLIP, which, after SAM segmentation, concatenate a text model CLIP to align the visual features of the segmented region with the text description to achieve automatic labeling.

[0046] This approach of cascading the SAM model and the classification model requires the classification model to generalize across all scenarios, typically requiring a large number of parameters and dense, extensive training data. On the one hand, the classification model has fixed categories and cannot be retrained to its own rules. On the other hand, excessively large classification parameters can lead to slow inference speeds and excessively high machine requirements, impacting the interactive annotation experience.

[0047] To solve the above problems, the applicant has developed a new semantic segmentation interactive annotation method for autonomous driving scenarios. This method does not require target differentiation, embedding complex classification models, or a large classification dataset. It can fine-tune its own dataset based on open source algorithms and simultaneously achieve semantic segmentation and classification functions.

[0048] This method is used to perform image segmentation and classification based on a source image and user interaction information, resulting in a segmented image in which target and non-target areas are separated and the target area is labeled with the target type. The user interaction information can be a point, box, or text input by the user.

[0049] Specifically, see Figure 1 , Figure 1It is a flow chart of an implementation method of the semantic segmentation interactive annotation method for the autonomous driving scenario of the present application.

[0050] like Figure 1 As shown, the method includes:

[0051] S100: Obtain image embedding and prompt embedding.

[0052] Among them, image embedding includes multi-scale features of the source image, and hint embedding includes the encoding vector of the interaction information.

[0053] Image embedding can be obtained by extracting multi-scale features from the source image based on an image encoder. For example, the image encoder can use Vision Transformer (ViT). In other implementations, the image encoder can also use other convolutional neural networks, which will not be repeated here.

[0054] Hint embedding is an embedded representation of user hints, which can include points, boxes, text, etc. The hint encoder can be used to convert the interaction information provided by the user into a vector aligned with the image features, which will not be described in detail here.

[0055] S200: Concatenate the prompt embedding and the initialized prediction token to obtain a token embedding.

[0056] Among them, the prediction token includes the mask token (mask_token) and the iou token (ious_token). The mask token is used to predict the mask result, and the iou token is used to predict the mask quality.

[0057] By concatenating the prompt embedding and the predicted token, we can obtain a token embedding that includes both sparse prompt mask information and predicted token.

[0058] S300, the token embedding and the image embedding are mapped and input into the transformer module, the token embedding is updated through the multi-head attention mechanism and the prompt embedding and the image embedding are fused to generate a context-aware intermediate representation.

[0059] The multi-head attention mechanism can fuse image features and prompt embeddings, enhance the feature response of the target area, model the global dependency between image features and prompt embeddings, generate context-aware intermediate representation src, and output the updated token embedding.

[0060] Specifically, in one embodiment, the transformer module can be a two-layer encoder, and each layer of the encoder first performs self-attention processing on the token embedding to achieve internal interaction of the token and eliminate prompt conflicts; then, cross-attention processing can be performed from the token embedding to the image embedding to inject the prompt semantics into the image features; then, new features are generated through a multi-layer perceptron to update each token; then, cross-attention processing is performed from the image embedding to the token embedding to feed back the image context to the token and correct the semantic ambiguity.

[0061] In order to update the position information in time, in each attention processing, the image embedding is superimposed with the position encoding once, and the token embedding is superimposed with the initial token embedding once.

[0062] In other implementations, the transformer module may also include encoders with other layers, which can also achieve the effects of this implementation.

[0063] S400: Input the updated mask token into the classification head, and predict the target type of the region of interest masked by the mask token.

[0064] The image features and hint embeddings are fused through S300, and the mask token and iou token used for prediction are updated. Among them, the mask token mask_token fuses the image features and hint embeddings through the Transformer module. Its essence is the embedded representation of the target instance, encoding the shape, position and local context information of the target.

[0065] Therefore, the mask token can be used as the input of the classification head, mapped to the category space, and the target type of the masked region of interest corresponding to the mask token is predicted, thereby realizing target type prediction without the need to cascade the classification model.

[0066] In one embodiment, the classification head may be a lightweight multi-layer perceptron (MLP) including one hidden layer, which can be implemented by adding a parallel classification branch at the end of the SAM decoder, thereby simultaneously achieving the functions of semantic segmentation and classification.

[0067] Furthermore, parallel to S400, mask output and quality evaluation are also included, including:

[0068] S500: Output the target mask based on the updated token embedding and context-aware intermediate representation.

[0069] The context-aware intermediate representation fuses image features with hint embeddings, and the updated token embedding includes the mask token mask_token for predicting the mask and the iou token ious_token for predicting the mask quality, so that the target mask can be combined to output.

[0070] Specifically, see Figure 2 , Figure 2 yes Figure 1 A flow chart of an implementation method corresponding to S500.

[0071] like Figure 2 As shown, the method of outputting the target mask includes:

[0072] S501: Input the updated mask token and the context-aware intermediate representation into the mask prediction head to generate multiple initial masks.

[0073] Specifically, the mask prediction head can perform dot product operations on the mask token and the context-aware intermediate representation to obtain multiple initial masks.

[0074] In one embodiment, in order to improve the detail restoration capability, shallow high-resolution features may be introduced, that is, high-resolution feature maps (high_res_features) are synchronously input into the mask prediction head and fused with deep features through skip connections.

[0075] The high-resolution feature map can be obtained by retaining the original high-resolution features of the image encoder.

[0076] S502: Based on the updated iou token, predict the intersection-over-union ratio of the initial mask and the true mask.

[0077] In parallel with generating the initial mask, the updated iou token is fed into the MLP to obtain the intersection-over-union (IoU) of each initial mask with the true mask, which is used to screen high-quality results.

[0078] S503: Embed the updated token into the input object scoring module to predict the confidence of the initial mask.

[0079] In parallel with generating the initial masks, the updated token embeddings are fed into the object score module (obj_score_), which predicts the confidence for each initial mask.

[0080] S504 : Sort the multiple initial masks based on the initial mask, the intersection-over-union ratio, and the confidence level, and output a target mask.

[0081] It can be understood that the initial masks can be sorted based on the intersection-over-union ratio and the confidence level, and then the target mask can be output.

[0082] In one embodiment, the target mask may include only the mask with the highest score. In another embodiment, the target mask may also include multiple initial masks sorted by score.

[0083] In order to improve the semantic consistency and boundary accuracy of masks through dynamic association of historical information, multi-instance decoupling and feature fusion, it also includes:

[0084] S505: Embed the updated token into the input object pointer projection layer to obtain the object pointer.

[0085] The object pointer Object Pointer is used to identify the location index of the target instance currently being processed in the memory library.

[0086] S506. Obtain the historical mask features stored in the memory bank based on the object pointer index, and perform cross-attention calculation with the target mask to correct the target mask.

[0087] By performing cross-attention calculations on historical mask features and the target mask, we can fill in local gaps in the target mask caused by occlusion or motion blur, thereby correcting the target mask. For example, in video segmentation, if the target in the current frame is occluded, the object pointer can recall the complete outline template from the historical features and generate a corrected target mask through weighted feature fusion.

[0088] S600 : Outputting the target type of the region of interest in the target mask based on the target mask and the target type of the region of interest in the mask corresponding to the mask token.

[0089] Based on the target mask and the target type of the masked region of interest corresponding to the mask token obtained in S400, the target type of the target masked region of interest can be output, thereby realizing an end-to-end multi-task model integrating segmentation and classification.

[0090] The methods based on the above-mentioned implementation methods do not require a huge classification dataset. By simply adding classification branches, one can fine-tune one's own dataset based on the open source algorithm, while realizing the functions of semantic segmentation and classification. This effectively solves the technical problems of the existing technology of the tandem SAM model and classification model, such as excessively large classification parameters, slow inference speed, high machine configuration requirements, and impact on the interactive annotation experience.

[0091] See also Figure 3 , Figure 3 This is a structural diagram of an implementation of the semantic segmentation interactive annotation model of this application. Figure 3 As shown, the image embedding (img features) and prompt embedding (prompt_embedding) are input into the transformer model for feature fusion, and the output token embedding (output) and the context-aware intermediate representation (src);

[0092] The intersection-over-union (iou_pred) of the predicted mask based on the ious token (ious_token);

[0093] Input the token embedding (output) into the object score module (obj_score_) to obtain the object score (obj_score_logits);

[0094] The mask token (mask_token) is input into the object pointer projection layer (obj_ptr_proj), classification head (class_head) and mask prediction head (mask prediction head) in parallel. The object pointer projection layer (obj_ptr_proj) maps the mask token to the object pointer (obj_ptr), and the classification head (class_head) outputs the target type of the masked region of interest; the mask prediction head (mask prediction head) outputs the initial mask (masks) based on the mask token (mask_token), context-aware intermediate representation (src) and high-resolution feature map (high_res_features), thereby realizing synchronous segmentation and classification of the image.

[0095] See also Figure 4 , Figure 4 This is a segmentation result diagram of an embodiment of the semantic segmentation interactive annotation method of the present application, such as Figure 4 As shown, the segmentation result of the method of the present application separates the target region of interest from other regions, and marks the target type with different colors, thereby achieving segmentation and labeling at the same time.

[0096] Specifically, Figure 4 Medium blue is the car, yellow is the pole, light blue is the sky, and non-interested areas are black ( Figure 4 Of course, in other embodiments, the target category may be labeled in a manner other than color, such as text or graphic, and the effects of this embodiment can be achieved.

[0097] This application also provides a semantic segmentation interactive annotation device for autonomous driving scenes, see Figure 5 , Figure 5 It is a structural diagram of an embodiment of a semantic segmentation interactive annotation device for an autonomous driving scenario of the present application.

[0098] like Figure 5 As shown, the device includes an acquisition module 21, a splicing module 22, a decoding module 23 and a mask classification module 24.

[0099] The acquisition module 21 is used to acquire image embedding and prompt embedding, the image embedding includes multi-scale features of the source image, and the prompt embedding includes an encoding vector of the interaction information;

[0100] The splicing module 22 is used to splice the prompt embedding and the initialized prediction token to obtain a token embedding. The prediction token includes a mask token and an iou token. The mask token is used to predict the mask result, and the iou token is used to predict the mask quality.

[0101] The decoding module 23 is used to map the token embedding and the image embedding into the transformer module, update the token embedding through the multi-head attention mechanism, and fuse the prompt embedding and the image embedding to generate a context-aware intermediate representation;

[0102] The mask classification module 24 is configured to input the updated mask tokens into the classification head, and predict the target type of the region of interest of the mask corresponding to the mask tokens.

[0103] In one embodiment, the apparatus further includes a mask generating module 25 and an output module 26 .

[0104] The mask generation module 25 is used to output a target mask based on the updated token embedding and context-aware intermediate representation;

[0105] The output module 26 is configured to output the target type of the region of interest in the target mask based on the target mask and the target type of the region of interest in the mask corresponding to the mask token.

[0106] As above Figures 1 to 4 , a method for semantic segmentation and interactive annotation of autonomous driving scenarios according to an embodiment of this specification is described. The details mentioned in the above description of the method embodiment are also applicable to the semantic segmentation and interactive annotation device for autonomous driving scenarios according to an embodiment of this specification. The above semantic segmentation and interactive annotation device for autonomous driving scenarios can be implemented using hardware, software, or a combination of hardware and software.

[0107] This application also provides an electronic device, see Figure 6 , Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of the present application. Figure 6 As shown, the electronic device 30 may include at least one processor 31, a memory 32 (e.g., a non-volatile memory), a storage 33, and a communication interface 34, and the at least one processor 31, the storage 32, the storage 33, and the communication interface 34 are connected together via an internal bus 35. The at least one processor 31 executes at least one computer-readable instruction stored or encoded in the storage 32.

[0108] It should be understood that the computer executable instructions stored in the memory 32, when executed, cause at least one processor 31 to perform the above combined operations in various embodiments of this specification. Figure 1-Figure 5 Describes the various operations and functions.

[0109] In the embodiments of the present specification, the electronic device 30 may include but is not limited to: a personal computer, a server computer, a workstation, a desktop computer, a laptop computer, a notebook computer, a mobile electronic device, a smart phone, a tablet computer, a cellular phone, a personal digital assistant (PDA), a handheld device, a messaging device, a wearable electronic device, a consumer electronic device, and the like.

[0110] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the above-mentioned elements implemented in software form), which, when executed by a machine, causes the machine to perform the above-mentioned combined embodiments of the present specification. Figure 1-Figure 5 Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes for implementing the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can be enabled to read and execute the instructions stored in the readable storage medium.

[0111] In this case, the program code itself read from the machine-readable medium can implement the functions of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0112] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0113] Those skilled in the art will appreciate that the various embodiments disclosed above may be modified and altered in various ways without departing from the essence of the invention. Therefore, the scope of protection of this specification shall be defined by the appended claims.

[0114] It should be noted that not all steps and units in the above processes and system structure diagrams are required, and certain steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or may be implemented by certain components in multiple independent devices.

[0115] In the above embodiments, hardware unit or module can be realized by mechanical means or electrical means. For example, a hardware unit, module or processor can include permanent dedicated circuit or logic (such as special processor, FPGA or ASIC) to complete the corresponding operation. Hardware unit or processor can also include programmable logic or circuit (such as general purpose processor or other programmable processor), can be temporarily set up to complete the corresponding operation by software. Concrete implementation (mechanical means or dedicated permanent circuit or temporary circuit) can be determined based on cost and time consideration.

[0116] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0117] The foregoing description of the present disclosure is provided to enable any person skilled in the art to implement or use the present disclosure. Various modifications to the present disclosure will be readily apparent to those skilled in the art, and the general principles herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is intended to be consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A semantic segmentation and interactive annotation method for autonomous driving scenes, characterized by: include: Obtaining an image embedding and a hint embedding, wherein the image embedding includes multi-scale features of the source image and the hint embedding includes an encoding vector of the interaction information; Concatenate the hint embedding and the initialized prediction token to obtain a token embedding, wherein the prediction token includes a mask token and an iou token, wherein the mask token is used to predict the mask result and the iou token is used to predict the mask quality; The token embedding and the image embedding are mapped and input into the transformer module, the token embedding is updated through a multi-head attention mechanism, and the prompt embedding and the image embedding are fused to generate a context-aware intermediate representation; The updated mask token is input into the classification head to predict the target type of the region of interest masked by the mask token.

2. The semantic segmentation interactive annotation method according to claim 1, characterized in that: Also includes: outputting a target mask based on the updated token embedding and the context-aware intermediate representation; Based on the target mask and the target type of the region of interest in the mask corresponding to the mask token, the target type of the region of interest in the target mask is output.

3. The semantic segmentation interactive annotation method according to claim 2, characterized in that: The step of outputting a target mask based on the updated token embedding and the context-aware intermediate representation comprises: Inputting the updated mask token and the context-aware intermediate representation into a mask prediction head to generate a plurality of initial masks; Based on the updated iou token, predict the intersection-over-union ratio of the initial mask and the true mask; Embedding the updated token into an input object score module to predict the confidence of the initial mask; Based on the initial mask, the intersection-over-union ratio, and the confidence level, a plurality of the initial masks are sorted, and a target mask is output.

4. The semantic segmentation interactive annotation method according to claim 3, characterized in that: In the step of inputting the updated mask token and the context-aware intermediate representation into a mask prediction head to generate a plurality of initial masks, the input of the mask prediction head further includes a high-resolution feature map.

5. The semantic segmentation interactive annotation method according to claim 3, characterized in that: The step of outputting a target mask based on the updated token embedding and the context-aware intermediate representation further comprises: Embed the updated token into the input object pointer projection layer to obtain an object pointer, where the object pointer is used to identify the location index of the target instance currently being processed in the memory bank; The historical mask features stored in the memory bank are obtained based on the object pointer index, and a cross-attention calculation is performed with the target mask to correct the target mask.

6. The semantic segmentation interactive annotation method according to claim 1, characterized in that: In the step of updating the token embedding and fusing the prompt embedding and the image embedding through a multi-head attention mechanism to generate a context-aware intermediate representation, The transformer module is a two-layer encoder, each layer of the encoder sequentially performs self-attention processing on the token embedding, performs cross-attention processing from the token embedding to the image embedding, generates new features through a multi-layer perceptron, and performs cross-attention processing from the image embedding to the token embedding; In each attention process, the image embedding is superimposed with the position encoding once, and the token embedding is superimposed with the initial token embedding once.

7. The semantic segmentation interactive annotation method according to claim 1, characterized in that: The classification head is a lightweight multi-layer perceptron including one hidden layer.

8. A semantic segmentation and interactive annotation device for autonomous driving scenes, characterized by: include: an acquisition module for acquiring an image embedding and a hint embedding, wherein the image embedding includes multi-scale features of the source image and the hint embedding includes an encoding vector of the interaction information; a splicing module for splicing the hint embedding and the initialized prediction token to obtain a token embedding, wherein the prediction token includes a mask token and an iou token, wherein the mask token is used to predict the mask result and the iou token is used to predict the mask quality; A decoding module is configured to input the mapping of the token embedding and the image embedding into a transformer module, update the token embedding through a multi-head attention mechanism, and fuse the prompt embedding and the image embedding to generate a context-aware intermediate representation; The mask classification module is used to input the updated mask token into the classification head and predict the target type of the region of interest masked by the mask token.

9. An electronic device comprising: at least one processor; as well as A memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to execute the semantic segmentation interactive annotation method according to any one of claims 1 to 7.

10. A machine-readable storage medium storing executable instructions, wherein when the instructions are executed, the machine executes the semantic segmentation interactive annotation method according to any one of claims 1 to 7.

Images