SAM-based self-prompting semantic segmentation method and apparatus, and storage medium

Abstract

The invention discloses a self-prompting semantic segmentation method and device based on SAM and a storage medium, and belongs to the technical field of image segmentation. The method comprises the following steps: acquiring a target image containing a target object; inputting the target image into a semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result comprises the position and the type of the defect of the target object; wherein the semantic segmentation model comprises an image encoder, a target detector, a prompt encoder and a mask decoder; the target detector is used for positioning the position area of the defect of the target object on the target image according to the image features output by the image encoder, and outputting a defect detection frame; the prompt encoder is used for encoding the defect detection frame and generating position prompt characteristics; and the mask decoder is used for fusing the image features and the position prompt features and generating a segmentation result of the target image. According to the invention, end-to-end industrial defect detection can be realized, and the efficiency of industrial defect detection is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of image segmentation, and in particular, relates to a self-prompted semantic segmentation method, device and storage medium based on SAM. Background Art

[0002] In the field of industrial manufacturing, with the continuous advancement of automation and intelligent technology, the requirements for product quality inspection are also increasing. Deep learning technology, especially semantic segmentation technology, is widely used in industrial defect detection due to its excellent performance in image recognition and target positioning.

[0003] In related technologies, after being trained with large-scale labeled data sets, semantic segmentation models can segment various objects on an image with the help of input prompt location information, thereby identifying product defects. However, semantic segmentation models usually rely on artificially given point prompts or box prompts to provide defect location information, which makes them difficult to apply to end-to-end industrial defect detection, greatly affecting the efficiency of industrial defect detection. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a self-prompted semantic segmentation method, device and storage medium based on SAM to achieve end-to-end industrial defect detection and improve the efficiency of industrial defect detection.

[0005] In a first aspect, the present application provides a self-prompted semantic segmentation method based on SAM, comprising:

[0006] Acquire a target image containing a target object;

[0007] Inputting the target image into a semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result includes the location and type of defects of the target object;

[0008] Among them, the semantic segmentation model includes an image encoder, a target detector, a hint encoder and a mask decoder; the target detector is used to locate the position area of ​​the defect of the target object on the target image according to the image features output by the image encoder, and output a defect detection box; the hint encoder is used to encode the defect detection box and generate a position hint feature; the mask decoder is used to fuse the image features and the position hint feature to generate a segmentation result of the target image.

[0009] According to the SAM-based self-prompt semantic segmentation method of the present application, a target image containing a target object is obtained; the target image is input into a semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result includes the location and type of the defect of the target object; wherein the semantic segmentation model includes an image encoder, a target detector, a prompt encoder and a mask decoder; the target detector is used to locate the location area of ​​the defect of the target object on the target image according to the image features output by the image encoder, and output a defect detection frame; the prompt encoder is used to encode the defect detection frame to generate a location prompt feature; the mask decoder is used to fuse the image feature and the location prompt feature to generate a segmentation result of the target image. The embodiment of the present application improves the semantic segmentation model by constructing a semantic segmentation model based on SAM, adds a target detector to the semantic segmentation model, the image encoder is responsible for extracting image features, and the target detector can locate the defect location area and output a defect detection frame, reducing the dependence on manual prompts, so that the model can automatically identify and locate defects from the image, realize end-to-end industrial defect detection, and improve the efficiency of industrial defect detection.

[0010] According to one embodiment of the present application, the image encoder is constructed using a lightweight convolutional neural network model.

[0011] In this embodiment, a lightweight convolutional neural network model is adopted as an image encoder. The lightweight convolutional neural network model reduces the number of parameters and the amount of calculation of the model by reducing the number of channels and using strategies such as depthwise separable convolution, thereby improving the reasoning speed and being able to better meet the requirements of adaptability to low-performance hardware.

[0012] According to one embodiment of the present application, the image encoder is trained according to the following method:

[0013] Acquire an original model based on SAM; the image encoder of the original model is constructed using a ViT-H model;

[0014] Constructing a training set; the images in the training set are annotated with defect labels;

[0015] Inputting the training set into the original SAM-based model to update the weights of the ViT-H model;

[0016] The ViT-H model with updated weights is used as the teacher model, and the lightweight convolutional neural network model is used as the student model, and knowledge distillation is performed to update the weights of the student model.

[0017] In this embodiment, the image encoder of the original model is built based on the ViT-H model. By training on a large-scale annotated dataset, the ViT-H model can learn rich image feature representations. The trained ViT-H model is used as the teacher model, and the lightweight convolutional neural network model is used as the student model to perform knowledge distillation, so that the student model can inherit effective feature extraction and defect recognition capabilities from the teacher model while keeping the model lightweight, reducing the computational complexity and inference delay of the model.

[0018] According to one embodiment of the present application, constructing a training set includes:

[0019] Inputting a plurality of sample images containing different target objects into the original model to obtain segmentation results for the sample images;

[0020] Generating annotations of the sample images according to adjustments made by the user to the segmentation results of the sample images to form an industrial data set;

[0021] The training set is constructed according to the industrial data set and a preset image segmentation data set.

[0022] In this embodiment, a plurality of sample images are input into the original model based on SAM to obtain preliminary segmentation results, and annotations are generated according to the user's adjustment of these segmentation results. The segmentation results of the original model and the user annotations are combined to reduce the dependence on a large amount of manual annotations. A training set is constructed by combining an industrial data set and a preset image segmentation data set. The training set not only contains rich industrial defect samples, but also integrates a variety of image segmentation scenarios, so that the model can learn the ability to identify and locate defects from different angles and under different conditions, thereby achieving higher efficiency and accuracy in actual industrial defect detection.

[0023] According to one embodiment of the present application, the step of inputting the training set into the original model based on SAM to update the weight of the ViT-H model includes:

[0024] Obtaining a prompt area selected by a user for an image in the training set;

[0025] Generate a prompt frame according to the prompt area, and select a preset number of points in the prompt frame as prompt points;

[0026] The prompt box and the prompt point are used as input to train the original model to update the weight of the ViT-H model; wherein, during a single iterative training process, the original model selects one of the prompt box and the prompt point to iterate, and during the training process, the original model alternately iterates the prompt box and the prompt point.

[0027] In this embodiment, by obtaining the prompt area selected by the user for the image in the training set, and generating a prompt box and a preset number of prompt points based on the prompt area, the model can focus on the key area in the image, thereby improving the accuracy of defect detection. The prompt box and prompt points are used as input to train the original model, and the prompt box and prompt points are alternately selected for iteration during a single iterative training process. This alternating iterative training strategy enables the model to learn the characteristics of defects from different angles and levels, thereby enhancing the generalization ability of the model.

[0028] According to one embodiment of the present application, the target detector is obtained by connecting the neck layer and the head layer of the yolov8 model in series.

[0029] In this embodiment, the target detector is constructed by adopting the neck layer and the head layer of the YOLOv8 model, so that the target detector can more effectively detect targets of different scales, thereby improving the accuracy of the defect detection frame.

[0030] According to one embodiment of the present application, the semantic segmentation model also includes a hint fine-tuning module, which is used to identify the edge of the target object at the target image so as to adjust the defect detection box output by the target detector according to the edge of the target image.

[0031] In this embodiment, by introducing a hint fine-tuning module, the model can dynamically adjust the defect detection frame according to the edge information of the target image, so that the hint information entering the mask decoder is more accurate, thereby improving the accuracy of the segmentation result.

[0032] In a second aspect, the present application provides a self-prompting semantic segmentation device based on SAM, comprising:

[0033] An acquisition module, used for acquiring a target image containing a target object;

[0034] An input module, used for inputting the target image into a semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result includes the location and type of defects of the target object;

[0035] Among them, the semantic segmentation model includes an image encoder, a target detector, a hint encoder and a mask decoder; the target detector is used to locate the position area of ​​the defect of the target object on the target image according to the image features output by the image encoder, and output a defect detection box; the hint encoder is used to encode the defect detection box and generate a position hint feature; the mask decoder is used to fuse the image features and the position hint feature to generate a segmentation result of the target image.

[0036] According to the SAM-based self-prompt semantic segmentation device of the present application, a target image containing a target object is obtained; the target image is input into a semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result includes the location and type of the defect of the target object; wherein the semantic segmentation model includes an image encoder, a target detector, a prompt encoder and a mask decoder; the target detector is used to locate the location area of ​​the defect of the target object on the target image according to the image features output by the image encoder, and output a defect detection frame; the prompt encoder is used to encode the defect detection frame to generate a location prompt feature; the mask decoder is used to fuse the image feature and the location prompt feature to generate a segmentation result of the target image. The embodiment of the present application improves the semantic segmentation model by constructing a SAM-based semantic segmentation model, adds a target detector to the semantic segmentation model, the image encoder is responsible for extracting image features, and the target detector can locate the defect location area and output a defect detection frame, reducing the dependence on manual prompts, so that the model can automatically identify and locate defects from the image, realize end-to-end industrial defect detection, and improve the efficiency of industrial defect detection.

[0037] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the SAM-based self-prompt semantic segmentation method as described in the first aspect above is implemented.

[0038] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the SAM-based self-prompted semantic segmentation method as described in the first aspect above.

[0039] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the SAM-based self-prompt semantic segmentation method as described in the first aspect above.

[0040] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the SAM-based self-prompted semantic segmentation method as described in the first aspect above.

[0041] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0042] According to the SAM-based self-prompt semantic segmentation method of the present application, a target image containing a target object is obtained; the target image is input into a semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result includes the location and type of the defect of the target object; wherein the semantic segmentation model includes an image encoder, a target detector, a prompt encoder and a mask decoder; the target detector is used to locate the location area of ​​the defect of the target object on the target image according to the image features output by the image encoder, and output a defect detection frame; the prompt encoder is used to encode the defect detection frame to generate a location prompt feature; the mask decoder is used to fuse the image feature and the location prompt feature to generate a segmentation result of the target image. The embodiment of the present application improves the semantic segmentation model by constructing a semantic segmentation model based on SAM, adds a target detector to the semantic segmentation model, the image encoder is responsible for extracting image features, and the target detector can locate the defect location area and output a defect detection frame, reducing the dependence on manual prompts, so that the model can automatically identify and locate defects from the image, realize end-to-end industrial defect detection, and improve the efficiency of industrial defect detection.

[0043] Furthermore, in some embodiments, by adopting a lightweight convolutional neural network model as an image encoder, the lightweight convolutional neural network model reduces the number of parameters and computational complexity of the model by reducing the number of channels, using depthwise separable convolution and other strategies, thereby improving the reasoning speed and being able to better meet the requirements of low-performance hardware adaptability.

[0044] Furthermore, in some embodiments, the image encoder of the original model is constructed based on the ViT-H model. By training on a large-scale annotated dataset, the ViT-H model can learn rich image feature representations. The trained ViT-H model is used as a teacher model, and the lightweight convolutional neural network model is used as a student model to perform knowledge distillation, so that the student model can inherit effective feature extraction and defect recognition capabilities from the teacher model while keeping the model lightweight, reducing the model's computational complexity and inference delay.

[0045] Furthermore, in some embodiments, by inputting multiple sample images into the original model based on SAM, preliminary segmentation results are obtained, and annotations are generated according to the user's adjustment of these segmentation results. The segmentation results of the original model and the user annotations are combined to reduce the dependence on a large amount of manual annotations. By combining the industrial data set and the preset image segmentation data set to construct a training set, the training set not only contains rich industrial defect samples, but also integrates a variety of image segmentation scenarios, so that the model can learn the ability to identify and locate defects from different angles and under different conditions, thereby achieving higher efficiency and accuracy in actual industrial defect detection.

[0046] Furthermore, in some embodiments, by obtaining the prompt area selected by the user for the image in the training set, and generating a prompt box and a preset number of prompt points based on the prompt area, the model can focus on the key areas in the image, thereby improving the accuracy of defect detection. The prompt box and prompt points are used as input to train the original model, and the prompt box and prompt points are alternately selected for iteration during a single iterative training process. This alternating iterative training strategy enables the model to learn the characteristics of defects from different angles and levels, thereby enhancing the generalization ability of the model.

[0047] Furthermore, in some embodiments, by using the neck layer and the head layer of the YOLOv8 model to construct an object detector, the object detector can more effectively detect objects of different scales, thereby improving the accuracy of the defect detection frame.

[0048] Furthermore, in some embodiments, by introducing a hint fine-tuning module, the model can dynamically adjust the defect detection frame according to the edge information of the target image, so that the hint information entering the mask decoder is more accurate, thereby improving the accuracy of the segmentation result.

[0049] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0051] Figure 1 It is a schematic diagram of the basic architecture of the SAM provided in the embodiment of the present application;

[0052] Figure 2 It is a flowchart of the SAM-based self-prompt semantic segmentation method provided in an embodiment of the present application;

[0053] Figure 3Schematic diagram of the architecture of the SAM-based semantic segmentation model provided in the embodiment of the present application;

[0054] Figure 4 This is one of the schematic diagrams of the training process of the image encoder provided in the embodiment of the present application;

[0055] Figure 5 This is one of the schematic diagrams of the training process of the image encoder provided in the embodiment of the present application;

[0056] Figure 6 is a schematic diagram of the training process of the target detector provided in an embodiment of the present application;

[0057] Figure 7 Schematic diagram of the training process of the prompt fine-tuning module provided in the embodiment of the present application;

[0058] Figure 8 It is a structural diagram of a self-prompting semantic segmentation device based on SAM provided in an embodiment of the present application;

[0059] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0061] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0062] Semantic segmentation is a deep learning technology that can assign each pixel in an image to a specific category label, thereby identifying and distinguishing different objects and regions in the image. Semantic segmentation technology is particularly suitable for scenarios that require accurate identification and location of objects in images, such as detecting defects on the surface of products on industrial production lines. By learning from a large amount of image data, the semantic segmentation model can identify various features in the image and associate these features with specific defect types, thereby achieving automated defect detection.

[0063] SAM (Segment Anything Model) is an advanced semantic segmentation model. A core feature of SAM is that it relies on manually given point prompts or box prompts to provide defect location information. Users can indicate the target that the model needs to segment through simple interactions, such as clicking on a certain area on the image. SAM can flexibly segment objects in the image based on these prompts without the need for special training for specific tasks or categories.

[0064] SAM's infrastructure Figure 1 As shown, it mainly includes three parts: Image Encoder, Prompt Encoder and Mask Decoder.

[0065] Image Encoder is responsible for processing the input image data and converting it into feature representations. These features contain basic information about objects in the image, such as shape, texture, and color. The image encoder can be a deep convolutional neural network that can capture multi-level features in the image.

[0066] The role of Prompt Encoder is to process the prompt information provided by the user. These prompts can be points, boxes or other forms of location information to indicate the objects that need to be segmented in the image. Prompt Encoder converts these prompts into feature vectors, which can be used to guide the segmentation process of the model so that the model can focus on the area specified by the user.

[0067] Mask Decoder is the core part of the model, which combines the features provided by Image Encoder and the prompt features generated by PromptEncoder to generate the final segmentation mask. Mask Decoder can accurately segment specific targets in the image based on the prompt information.

[0068] The workflow of SAM is as follows: the input image is sent to Image Encoder to extract image features. The user enters prompt information, which is sent to Prompt Encoder and converted into prompt features. The image features and prompt features are then sent to Mask Decoder, which generates a segmentation mask based on the image features and prompt features, thereby achieving accurate segmentation of the target in the image.

[0069] However, SAM usually relies on manually given point prompts or box prompts to provide defect location information, which is difficult to apply to end-to-end industrial defect detection, greatly affecting the efficiency of industrial defect detection.

[0070] In the following, in conjunction with the accompanying drawings, the SAM-based self-prompt semantic segmentation method, device and storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0071] Among them, the SAM-based self-prompt semantic segmentation method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0072] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0073] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0074] The SAM-based self-prompt semantic segmentation method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the SAM-based self-prompt semantic segmentation method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, wearable devices, etc. The SAM-based self-prompt semantic segmentation method provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.

[0075] like Figure 2 As shown, the SAM-based self-prompt semantic segmentation method includes: step 210 and step 220.

[0076] Step 210: Acquire a target image containing a target object.

[0077] In the embodiment of the present application, in industrial inspection, the target objects may include different types of products, such as electronic products, auto parts, textiles, food, etc. The target images of these products can be obtained by taking pictures with a high-definition camera or an industrial camera.

[0078] Step 220: Input the target image into the SAM-based semantic segmentation model to obtain a segmentation result of the target image; the segmentation result includes the location and type of the defect of the target object.

[0079] In the embodiment of the present application, the architecture of the semantic segmentation model based on SAM is as follows: Figure 3 As shown, the semantic segmentation model can include an image encoder, an object detector, a hint encoder, and a mask decoder.

[0080] In some embodiments, the image encoder can be used to map the input target image to the image feature space to obtain the image features of the target image. In the SAM architecture, the image encoder can use pre-trained visual encoders, such as Vit (Vision Transformer), ViT-H (Vision Transformer-Huge), etc., to extract image features.

[0081] In some embodiments, in the SAM model in the related art, ViT-H is usually used as the image encoder, but the computational complexity of ViT-H is large and the hardware requirements are too high. Therefore, a lightweight convolutional neural network (CNN) model can be used to construct an image encoder, such as RepVit (Repeating Vision Transformer), MobileNetV2, EfficientNet, etc.

[0082] In this embodiment, a lightweight convolutional neural network model is adopted as an image encoder. The lightweight convolutional neural network model reduces the number of parameters and the amount of calculation of the model by reducing the number of channels and using strategies such as depthwise separable convolution, thereby improving the reasoning speed and being able to better meet the requirements of adaptability to low-performance hardware.

[0083] In some embodiments, the target detector may be a model based on deep learning, such as the YOLO (You Only Look Once) series of algorithms, Faster R-CNN (Faster Region-based Convolutional Neural Network), etc. The target detector may be used to locate the location area of ​​the defect of the target object on the target image according to the image features output by the image encoder, and output a defect detection frame. The defect detection frame may include one or more rectangular bounding boxes for identifying the location area of ​​the defect of the target object on the target image.

[0084] In some embodiments, the hint encoder is used to encode the defect detection box to generate a position hint feature. Specifically, the hint encoder can use a small Transformer encoder network to encode the hint information into a position hint feature vector. The input hint information may include point coordinates, bounding boxes, and low-resolution masks, etc. The hint encoder can use position encoding to represent points and boxes.

[0085] In some embodiments, the mask decoder can be used to fuse image features and position hint features to generate a segmentation result of the target image. Specifically, the mask decoder can use the Transformer architecture to process the input image features and position hint features. The mask decoder fuses image features and position hint features through a self-attention mechanism and a cross-attention mechanism, generates a representation of the mask, and uses a classifier to calculate the foreground mask probability of each position in the target image, and outputs the segmentation result. Among them, the segmentation result may include one or more binary segmentation masks, each segmentation mask corresponds to a defect of the target object in the target image, and marks the location and type of the defect. For example, if the target object is a mobile phone screen, the segmentation result may include displaying the scratch position, depression position, etc. on the mobile phone screen on the target image, and marking these positions as "scratches" and "depressions".

[0086] According to the SAM-based self-prompt semantic segmentation method of the present application, a target image containing a target object is obtained; the target image is input into a SAM-based semantic segmentation model to obtain a segmentation result of the target image; the segmentation result includes the location and type of the defect of the target object; wherein the semantic segmentation model includes an image encoder, a target detector, a prompt encoder and a mask decoder; the target detector is used to locate the location area of ​​the defect existing in the target object on the target image according to the image features output by the image encoder, and output a defect detection frame; the prompt encoder is used to encode the defect detection frame and generate a location prompt feature; the mask decoder is used to fuse the image features and the location prompt feature to generate a segmentation result of the target image. The embodiment of the present application improves the semantic segmentation model by constructing a SAM-based semantic segmentation model, adds a target detector to the semantic segmentation model, the image encoder is responsible for extracting image features, and the target detector can locate the defect location area and output a defect detection frame, reducing the dependence on manual prompts, so that the model can automatically identify and locate defects from the image, realize end-to-end industrial defect detection, and improve the efficiency of industrial defect detection.

[0087] In some embodiments, the image encoder is trained according to the following method:

[0088] Obtain an original model based on SAM; the image encoder of the original model is constructed using the ViT-H model;

[0089] Construct a training set; images in the training set are annotated with defect labels;

[0090] Input the training set into the original SAM-based model to update the weights of the ViT-H model;

[0091] The ViT-H model with updated weights is used as the teacher model, and the lightweight convolutional neural network model is used as the student model. Knowledge distillation is performed to update the weights of the student model.

[0092] In this embodiment, since the official does not provide a lightweight convolutional neural network model as the relevant pre-trained weights of the image encoder, the teacher model ViT-H can be obtained through training first, and then the information of the teacher model can be distilled into the lightweight convolutional neural network model.

[0093] In this embodiment, a SAM-based original model can be obtained, the image encoder part of the original model is constructed using the ViT-H model, and the original model has been pre-trained on a large-scale dataset and has a strong feature extraction capability.

[0094] In this embodiment, a training set containing images of different target objects can be constructed, and these images are annotated with defect labels, and the defect labels include the defect location and defect type of the target object in the image. The training set can be used to train the original model to fine-tune the weights of the ViT-H model. During the training process, the parameters of the image encoder part of the original model can be set to an updateable state, and the parameters of the prompt encoder and the mask decoder can be set to a state that does not allow updating. A preset round of iterative training can be performed to obtain the weights of the fine-tuned ViT-H model.

[0095] In some embodiments, constructing a training set includes:

[0096] Input multiple sample images containing different target objects into the original model to obtain segmentation results for the sample images;

[0097] Generate annotations of sample images based on the user's adjustment of the segmentation results of sample images to form an industrial data set;

[0098] Construct training sets based on industrial datasets and preset image segmentation datasets.

[0099] In this embodiment, a series of sample images can be collected first. Different sample images can include different or the same target objects, and these target objects cover different defects. The sample images are input into the original model to obtain the segmentation results of the sample images. The segmentation results include the location bounding box and defect type indicating the location of the defect in the sample image. After obtaining the segmentation results, the user can review and adjust the segmentation results to generate annotations for the sample images to form an industrial data set. For example, the user can check the segmentation results provided by the original model and adjust them according to actual needs. These adjustments can include correcting the position of the bounding box, adjusting the accuracy of the segmentation mask, or correcting the type and position of the defect, etc. Accurate annotations can be generated based on the user's adjustments to guide the training of the model.

[0100] In this embodiment, in order to improve the generalization ability of the model, a preset image segmentation dataset, such as the SA-1B (Segment Anything 1Billion) dataset, can also be combined to generate a final training set. In one example, the top 1% of SA-1B images and the industrial dataset can be selected as the training set, and 25% of the samples are extracted from the training set as an independent validation set.

[0101] In this embodiment, a plurality of sample images are input into the original model based on SAM to obtain preliminary segmentation results, and annotations are generated according to the user's adjustment of these segmentation results. The segmentation results of the original model and the user annotations are combined to reduce the dependence on a large amount of manual annotations. A training set is constructed by combining an industrial data set and a preset image segmentation data set. The training set not only contains rich industrial defect samples, but also integrates a variety of image segmentation scenarios, so that the model can learn the ability to identify and locate defects from different angles and under different conditions, thereby achieving higher efficiency and accuracy in actual industrial defect detection.

[0102] In some embodiments, the training set is input into the original SAM-based model to update the weights of the ViT-H model, including:

[0103] Obtain the hint area selected by the user for the image in the training set;

[0104] Generate a prompt box according to the prompt area, and select a preset number of points inside the prompt box as prompt points;

[0105] The prompt box and prompt point are used as input to train the original model to update the weight of the ViT-H model; wherein, during a single iteration training process, the original model selects one of the prompt box and the prompt point to iterate, and during the training process, the original model alternately iterates the prompt box and the prompt point.

[0106] In this embodiment, the training process is as follows Figure 4 As shown in the figure. During the training process, a single iteration training can obtain a prompt area (mask area) from the sample image of the training set. The prompt area can be an area pre-marked by the user, and the prompt area can be used as prompt information to indicate the target to be segmented in the image. Prompt Encoder converts these prompts into feature vectors, which can be used to guide the segmentation process of the model so that the model can focus on the area specified by the user.

[0107] A rectangular prompt box can be generated according to the prompt area. A preset number of points can be selected in the promotion box as prompt points. For example, 1-6 points can be selected as prompt points. In the subsequent training process, the prompt points and the prompt box can be used as inputs of the Prompt Encoder.

[0108] In this embodiment, only one of the prompt box and the prompt point is selected as the prompt information in a single iteration. During the entire training process, the prompt box and the prompt point are alternately rotated. For example, the prompt box is selected as the prompt information in this iteration, and the prompt point is selected as the prompt information in the next iteration, and so on. Specifically, at the beginning of training, the original image in the training set will pass through the image encoder based on the ViT-H model. The image encoder extracts features and outputs image features to the mask decoder and the prompt encoder. The prompt encoder can also accept additional prompt information, such as a prompt box or a prompt point. The prompt encoder generates a position prompt feature based on the image features and the prompt information, and inputs the position prompt feature to the decoding encoder. The decoding encoder generates a segmentation result based on the image features and the position prompt feature, compares the segmentation result with the original image label, performs back propagation based on the difference between the segmentation result and the original image label, adjusts the weight of the ViT-H model through the optimization algorithm, and continues the next round of training.

[0109] In this embodiment, by obtaining the prompt area selected by the user for the image in the training set, and generating a prompt box and a preset number of prompt points based on the prompt area, the model can focus on the key area in the image, thereby improving the accuracy of defect detection. The prompt box and prompt points are used as input to train the original model, and the prompt box and prompt points are alternately selected for iteration during a single iterative training process. This alternating iterative training strategy enables the model to learn the characteristics of defects from different angles and levels, thereby enhancing the generalization ability of the model.

[0110] In this embodiment, after the weight update of the ViT-H model is completed, the knowledge distillation method can be used to use the trained and weight-updated ViT-H model as the teacher model, and select a lightweight convolutional neural network model, such as repvit-v2, as the student model. The role of the teacher model is to provide a performance benchmark, and its output can be used as the target of the student model training. The student model uses a lightweight convolutional neural network model, so that it can adapt to low-performance hardware and improve reasoning efficiency.

[0111] like Figure 5As shown, taking the lightweight convolutional neural network model repvit-v2 as an example, the industrial dataset and the 1% SA-1B dataset can be used as training sets for the training and distillation process, so that repvit-v2 can learn the characteristics of industrial scenarios. The images of the training set can be input into the image encoder based on ViT-H, and the image encoder based on ViT-H can generate image embeddings for each image. These image embeddings contain the feature representation of the image. The images of the training set are also input into the image encoder based on repvit-v2, and the image encoder based on repvit-v2 can also generate image embeddings for each image.

[0112] During the knowledge distillation process, the goal is to minimize the difference between the output features of the teacher model and the student model. For example, the cross entropy loss can be used to calculate the difference between the output features of the teacher model and the student model. Through backpropagation and optimization algorithms (such as gradient descent), the cross entropy loss is used to update the weights of the student model repvit-v2. After multiple training iterations, the student model can learn the feature representation of the teacher model, thereby improving the performance of the student model on industrial images. After knowledge distillation, the student model repvit-v2 obtains the industrial pre-trained weights, making the repvit-v2-based image encoder more effective when processing industrial images.

[0113] In this embodiment, the image encoder of the original model is built based on the ViT-H model. By training on a large-scale annotated dataset, the ViT-H model can learn rich image feature representations. The trained ViT-H model is used as the teacher model, and the lightweight convolutional neural network model is used as the student model to perform knowledge distillation, so that the student model can inherit effective feature extraction and defect recognition capabilities from the teacher model while keeping the model lightweight, reducing the computational complexity and inference delay of the model.

[0114] In some embodiments, the target detector is obtained by connecting the neck layer and the head layer of the yolov8 model in series.

[0115] In this embodiment, the neck layer of the yolov8 model is mainly responsible for multi-scale feature fusion, which enhances the feature representation capability by fusing feature maps from different stages of Backbone. YOLOv8 adopts the PAN-FAN (Path Aggregation Network with Feature Aggregation Network) structure in the Neck part, including two PAN modules and one SPPF (Spatial Pyramid Pooling) module. The PAN module enhances the expression capability of feature maps through bottom-up and top-down path aggregation. The SPPF module is used for pooling operations of different scales, splicing feature maps of different scales together to improve the detection capability of targets of different sizes.

[0116] The head layer is responsible for the final target detection and classification tasks, including a detection head and a classification head. The detection head contains a series of convolutional layers and deconvolutional layers to generate detection results, including predicted bounding boxes, categories, and confidence levels. The classification head uses global average pooling to classify each feature map and outputs the probability distribution of each category. YOLOv8 introduces the Decoupled-Head idea in the Head part, making the network training and reasoning more efficient.

[0117] In the object detector, the features output by the Neck layer are directly sent to the Head layer. The features fused and enhanced by the Neck layer are processed in the Head layer, and the detection results, including the bounding box, category, and confidence of the object, are output. By connecting the Neck layer and the Head layer in series, the object detector can more effectively utilize feature information of different scales and improve the accuracy of object detection.

[0118] In this embodiment, the object detector also needs to be trained to improve the output capability of the defect detection frame. The training process is as follows: Figure 6 As shown, the above training set can also be used for training. The images of the training set can first be input into the image encoder based on repvit-v2. The image features extracted from the repViT-v2 image encoder can be upsampled to increase the resolution of the image features and restore the spatial details of the image features.

[0119] The upsampled image features are then input to the Neck layer of YOLOv8, which further processes the image features, such as feature fusion and multi-scale feature extraction, to improve the accuracy of detection. The image features processed by the Neck layer are input to the Head layer of YOLOv8, which is responsible for the final target detection task, including bounding box regression and category classification, and outputs defect detection boxes that identify the location and size of the defects of the target object in the image. According to the difference between the output defect detection box and the label, the weights of the Neck layer and the Head layer are updated through back propagation and optimization algorithms.

[0120] In this embodiment, the target detector is constructed by adopting the neck layer and the head layer of the YOLOv8 model, so that the target detector can more effectively detect targets of different scales, thereby improving the accuracy of the defect detection frame.

[0121] In some embodiments, the semantic segmentation model further includes a hint fine-tuning module, which is used to identify the edge of the target object at the target image so as to adjust the defect detection box output by the target detector according to the edge of the target image.

[0122] In this embodiment, in order to further improve the accuracy of the defect detection box output by the target detector, the semantic segmentation model may also include a prompt fine-tuning module (adaptor module), which may use some fully connected layers or small sub-networks to adjust the defect detection box output by the target detector.

[0123] The hint fine-tuning module can identify the edge of the target object by analyzing image features and using edge detection algorithms such as Canny edge detection or Sobel operator, and output edge recognition results. The hint fine-tuning module can use the edge recognition results to adjust the position of the defect detection box generated by the object detector. Or the hint fine-tuning module can input the edge recognition results to the mask decoder, and the mask decoder can adjust the position of the defect detection box generated by the object detector according to the edge recognition results.

[0124] In this embodiment, the prompt fine-tuning module also needs to be trained to improve the adjustment ability of the defect detection frame. Figure 7As shown in the figure, during the training process, the hint fine-tuning module first calculates the overall maximum circumscribed rectangle of the defect area based on the label of the original image as the frame hint feature, and encodes it by the hint encoder to obtain the frame hint feature. In addition, the hint fine-tuning module encodes the label of the original image to obtain the mask hint feature, and uses the canny operator to obtain the edge contour of the label of the original image to obtain a binary image of the edge contour with the same size as the original image, and aligns the encoding to obtain the edge contour mask hint feature, and then splices it with the previously obtained mask hint feature in the channel direction, and finally inputs it into the decoder together with the frame hint feature.

[0125] Specifically, Figure 7 As shown, the original image can be a sample image in the above training set. The training process can use an image encoder based on repvit-v2. The bbox sparse emb represents the box hint feature generated by the hint encoder according to the label of the original image, the mask dense emb represents the mask hint feature generated by the hint fine-tuning module according to the dense embedding of the label of the original image, and the edge map dense emb represents the edge contour mask hint feature generated by the hint fine-tuning module using the Canny operator to extract the edge contour information from the original image label. The mask decoder receives the box hint feature from the hint encoder, the mask hint feature and the edge contour mask hint feature from the hint fine-tuning module, and the image feature from the image encoder, and outputs the segmentation result. According to the difference between the output segmentation result and the label, the weight of the hint fine-tuning module is updated through back propagation and optimization algorithms.

[0126] In this embodiment, by introducing a hint fine-tuning module, the model can dynamically adjust the defect detection frame according to the edge information of the target image, so that the hint information entering the mask decoder is more accurate, thereby improving the accuracy of the segmentation result.

[0127] In order to verify the segmentation effect of the self-prompted semantic segmentation method based on SAM provided in the embodiment of the present application, the performance of the present application was verified on 6 sets of industrial scene data sets, and the conventional semantic segmentation model segformer-b0 was used as the performance baseline. The experiment was carried out in the hardware environment: graphics card: 3090, CPU: Intel (R) Xeon (R) Platinum 8358CPU @ 2.60GHz, the training image input size was fixed to (1024, 1024), each data set was trained twice using the complete training set and validation set, and the models trained twice were inferred, and the average of the results was finally taken.

[0128] The specific experimental results are shown in Table 1 and Table 2.

[0129] Table 1

[0130]

[0131] Table 2

[0132]

[0133] Among them, Table 1 is a comparison of the accuracy performance of the SAM-based semantic segmentation model provided in the embodiment of the present application and the conventional semantic segmentation model, and Table 2 is a comparison of the reasoning efficiency and video memory occupancy of the SAM-based semantic segmentation model provided in the embodiment of the present application and the conventional semantic segmentation model.

[0134] It can be seen from the results given in Tables 1 and 2 that the SAM-based semantic segmentation model provided in the embodiments of the present application can effectively exert the SAM accuracy advantage in end-to-end industrial defect detection tasks, while meeting the requirements of practical applications for model reasoning efficiency and low-performance hardware adaptability, and ultimately achieve project cost reduction and efficiency improvement.

[0135] The SAM-based self-prompting semantic segmentation method provided in the embodiment of the present application can be executed by a SAM-based self-prompting semantic segmentation device. In the embodiment of the present application, a SAM-based self-prompting semantic segmentation method is executed by a SAM-based self-prompting semantic segmentation device as an example to illustrate the SAM-based self-prompting semantic segmentation device provided in the embodiment of the present application.

[0136] The embodiment of the present application also provides a self-prompting semantic segmentation device based on SAM.

[0137] like Figure 8 As shown, the SAM-based self-prompt semantic segmentation device includes:

[0138] An acquisition module 810 is used to acquire a target image containing a target object;

[0139] An input module 820 is used to input the target image into the semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result includes the location and type of the defect of the target object;

[0140] Among them, the semantic segmentation model includes an image encoder, an object detector, a hint encoder and a mask decoder; the object detector is used to locate the position area of ​​the defect of the target object on the target image according to the image features output by the image encoder, and output the defect detection box; the hint encoder is used to encode the defect detection box and generate the position hint features; the mask decoder is used to fuse the image features and the position hint features to generate the segmentation result of the target image.

[0141] According to the SAM-based self-prompt semantic segmentation device of the present application, a target image containing a target object is obtained; the target image is input into a semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result includes the location and type of the defect of the target object; wherein the semantic segmentation model includes an image encoder, a target detector, a prompt encoder and a mask decoder; the target detector is used to locate the location area of ​​the defect existing in the target object on the target image according to the image features output by the image encoder, and output a defect detection frame; the prompt encoder is used to encode the defect detection frame and generate a location prompt feature; the mask decoder is used to fuse the image features and the location prompt feature to generate a segmentation result of the target image. The embodiment of the present application improves the semantic segmentation model by constructing a semantic segmentation model based on SAM, adds a target detector to the semantic segmentation model, the image encoder is responsible for extracting image features, and the target detector can locate the defect location area and output the defect detection frame, reducing the dependence on manual prompts, so that the model can automatically identify and locate defects from the image, realize end-to-end industrial defect detection, and improve the efficiency of industrial defect detection.

[0142] In some embodiments, the image encoder is constructed using a lightweight convolutional neural network model.

[0143] In some embodiments, the image encoder is trained according to the following method:

[0144] Obtain an original model based on SAM; the image encoder of the original model is constructed using the ViT-H model;

[0145] Construct a training set; images in the training set are annotated with defect labels;

[0146] Input the training set into the original SAM-based model to update the weights of the ViT-H model;

[0147] The ViT-H model with updated weights is used as the teacher model, and the lightweight convolutional neural network model is used as the student model. Knowledge distillation is performed to update the weights of the student model.

[0148] In some embodiments, constructing a training set includes:

[0149] Input multiple sample images containing different target objects into the original model to obtain segmentation results for the sample images;

[0150] Generate annotations of sample images based on the user's adjustment of the segmentation results of sample images to form an industrial data set;

[0151] Construct training sets based on industrial datasets and preset image segmentation datasets.

[0152] In some embodiments, the training set is input into the original SAM-based model to update the weights of the ViT-H model, including:

[0153] Obtain the hint area selected by the user for the image in the training set;

[0154] Generate a prompt box according to the prompt area, and select a preset number of points inside the prompt box as prompt points;

[0155] The prompt box and prompt point are used as input to train the original model to update the weight of the ViT-H model; wherein, during a single iteration training process, the original model selects one of the prompt box and the prompt point to iterate, and during the training process, the original model alternately iterates the prompt box and the prompt point.

[0156] In some embodiments, the target detector is obtained by connecting the neck layer and the head layer of the yolov8 model in series.

[0157] In some embodiments, the semantic segmentation model further includes a hint fine-tuning module, which is used to identify the edge of the target object at the target image so as to adjust the defect detection box output by the target detector according to the edge of the target image.

[0158] The self-prompt semantic segmentation device based on SAM in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0159] The SAM-based self-prompt semantic segmentation device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0160] In some embodiments, Fig. 9 As shown, an embodiment of the present application also provides an electronic device 900, including a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901. When the program is executed by the processor 901, each process of the above-mentioned SAM-based self-prompt semantic segmentation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0161] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0162] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned SAM-based self-prompt semantic segmentation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0163] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0164] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned SAM-based self-prompt semantic segmentation method.

[0165] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0166] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned SAM-based self-prompt semantic segmentation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0167] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0168] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0170] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0171] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0172] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A self-prompted semantic segmentation method based on SAM, characterized in that: include: Acquire a target image containing a target object; Inputting the target image into a semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result includes the location and type of defects of the target object; Among them, the semantic segmentation model includes an image encoder, a target detector, a hint encoder and a mask decoder; the target detector is used to locate the position area of ​​the defect of the target object on the target image according to the image features output by the image encoder, and output a defect detection box; the hint encoder is used to encode the defect detection box and generate a position hint feature; the mask decoder is used to fuse the image features and the position hint feature to generate a segmentation result of the target image.

2. The method according to claim 1, characterized in that The image encoder is constructed using a lightweight convolutional neural network model.

3. The method according to claim 2, characterized in that The image encoder is trained according to the following method: Acquire an original model based on SAM; the image encoder of the original model is constructed using a ViT-H model; Constructing a training set; the images in the training set are annotated with defect labels; Inputting the training set into the original SAM-based model to update the weights of the ViT-H model; The ViT-H model with updated weights is used as the teacher model, and the lightweight convolutional neural network model is used as the student model, and knowledge distillation is performed to update the weights of the student model.

4. The method according to claim 3, characterized in that The constructing of the training set comprises: Inputting a plurality of sample images containing different target objects into the original model to obtain segmentation results for the sample images; Generating annotations of the sample images according to adjustments made by the user to the segmentation results of the sample images to form an industrial data set; The training set is constructed according to the industrial data set and a preset image segmentation data set.

5. The method according to claim 3, characterized in that: The step of inputting the training set into the original model based on SAM to update the weight of the ViT-H model comprises: Obtaining a prompt area selected by a user for an image in the training set; Generate a prompt frame according to the prompt area, and select a preset number of points in the prompt frame as prompt points; The prompt box and the prompt point are used as input to train the original model to update the weight of the ViT-H model; wherein, during a single iterative training process, the original model selects one of the prompt box and the prompt point to iterate, and during the training process, the original model alternately iterates the prompt box and the prompt point.

6. The method according to claim 1, characterized in that The target detector is obtained by connecting the neck layer and the head layer of the yolov8 model in series.

7. The method according to claim 1, characterized in that The semantic segmentation model also includes a hint fine-tuning module, which is used to identify the edge of the target object in the target image so as to adjust the defect detection box output by the target detector according to the edge of the target image.

8. A self-prompted semantic segmentation device based on SAM, characterized in that: include: An acquisition module, used for acquiring a target image containing a target object; An input module, used for inputting the target image into a semantic segmentation model based on SAM to obtain a segmentation result of the target image; the segmentation result includes the location and type of defects of the target object; Among them, the semantic segmentation model includes an image encoder, a target detector, a hint encoder and a mask decoder; the target detector is used to locate the position area of ​​the defect of the target object on the target image according to the image features output by the image encoder, and output a defect detection box; the hint encoder is used to encode the defect detection box and generate a position hint feature; the mask decoder is used to fuse the image features and the position hint feature to generate a segmentation result of the target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images