Semantic segmentation method and device based on SAM model, equipment and storage medium

Abstract

The invention relates to the field of image processing and digital medical treatment, and discloses a semantic segmentation method and device based on an SAM model, equipment and a storage medium, and the method comprises the steps: generating at least one group of semantic tag images based on a preset semantic segmentation model and an initial image; determining at least one image segmentation mask of the initial image through an SAM model and the initial image; and based on each image segmentation mask and each semantic tag image, determining a target semantic feature of the initial image. Through the above mode, by using the SAM model, the generated semantic tag image not only provides a category tag for each pixel in the image, but also realizes accurate classification of each pixel in the image, and provides high-resolution semantic information for image analysis. By automatically determining the image segmentation mask and the semantic features, the demand of manual feature engineering is reduced, and the accuracy of the semantic segmentation data set is improved.

Description

Technical Field

[0001] The present application relates to the fields of image processing and digital medicine, and in particular to a semantic segmentation method, apparatus, device and storage medium based on a SAM model. Background Art

[0002] In the field of digital medicine, semantic segmentation can be used for organ segmentation in medical images, such as the heart, lungs, liver, etc. By accurately segmenting the organ regions, it can help doctors diagnose diseases, plan surgeries, and specify treatment plans. The general segmentation model can also be used for image analysis in the general digital financial field due to its strong generalization, such as credit card image segmentation in credit card fraud detection, check image segmentation in check recognition, etc. By accurately segmenting the region of interest, key image features and information can be extracted for risk assessment and fraud detection.

[0003] Semantic Segmentation: Semantic segmentation combines image classification, object detection and image segmentation. The specific task is to assign a semantic label to each pixel in the image. Pixels belonging to the same category should be classified into one category. Therefore, semantic segmentation understands the image at the pixel level. The task of video semantic segmentation is to assign a semantic label to each pixel in each frame of the video.

[0004] SAM (Segment Anything Model) is a highly generalizable object segmentation algorithm that can provide accurate masks. The dataset SA-1B used by SAM is the largest image segmentation dataset to date, providing detailed mask segmentation annotations. However, neither SAM nor SA-1B provides category predictions or annotations for each mask. This makes it difficult for researchers to use the powerful SAM algorithm to directly solve semantic segmentation tasks or use SA-1B to train their own models. Other semantic segmentation models (such as Segformer, Maskformer, etc.) or image caption methods (such as BL IP) can provide rich semantic annotations. However, their mask segmentation prediction results are not as comprehensive and accurate as those generated by SAM, and the mask edges generated by SAM are more accurate. Therefore, how to semantically segment images through the SAM model and improve the accuracy of semantic segmentation datasets has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a semantic segmentation method, apparatus, device and storage medium based on the SAM model to improve the accuracy of semantic segmentation data sets.

[0006] In a first aspect, the present application provides a semantic segmentation method based on a SAM model, the method comprising:

[0007] Based on a preset semantic segmentation model and an initial image, generating at least one set of semantically labeled images;

[0008] Determine at least one image segmentation mask of the initial image by using a preset image segmentation model SAM model and the initial image;

[0009] Based on each of the image segmentation masks and each of the semantic label images, a target semantic feature of the initial image is determined.

[0010] In a second aspect, the present application further provides a semantic segmentation device based on the SAM model, the device comprising:

[0011] Acquire training set data and verification set data, wherein the training set data includes training images and training labels, and the verification set data includes verification images and verification labels;

[0012] The initial SAM model is trained through each of the training images and each of the training labels, and the initial SAM model is iteratively verified through each of the verification images and each of the verification labels to determine the preset SAM model.

[0013] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the semantic segmentation method based on the SAM model as described above when executing the computer program.

[0014] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the semantic segmentation method based on the SAM model as described above.

[0015] The present application discloses a semantic segmentation method, apparatus, device and storage medium based on a SAM model. The semantic segmentation method based on the SAM model includes generating at least one set of semantically labeled images based on a preset semantic segmentation model and an initial image; determining at least one image segmentation mask of the initial image through a preset image segmentation model SAM model and the initial image; and determining the target semantic features of the initial image based on each of the image segmentation masks and each of the semantically labeled images. In the above manner, the semantically labeled images generated by the present application using the SAM model not only provide category labels for each pixel in the image, but also achieve accurate classification of each pixel in the image, and provide high-resolution semantic information for image analysis. By automatically determining image segmentation masks and semantic features, the need for manual feature engineering is reduced, and the accuracy of semantic segmentation datasets is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a schematic flow chart of a semantic segmentation method based on a SAM model provided in the first embodiment of the present application;

[0018] Figure 2 is a schematic flow chart of a semantic segmentation method based on a SAM model provided in the second embodiment of the present application;

[0019] Figure 3 is a schematic flow chart of a semantic segmentation method based on a SAM model provided in the third embodiment of the present application;

[0020] Figure 4 A schematic block diagram of a semantic segmentation device based on a SAM model provided in an embodiment of the present application;

[0021] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0024] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] The embodiments of the present application provide a semantic segmentation method, apparatus, device and storage medium based on the SAM model. Among them, the semantic segmentation method based on the SAM model can be applied to a server. By using the SAM model, the generated semantic label image not only provides a category label for each pixel in the image, but also realizes accurate classification of each pixel in the image, and provides high-resolution semantic information for image analysis. By automatically determining the image segmentation mask and semantic features, the need for manual feature engineering is reduced, and the accuracy of the semantic segmentation data set is improved. Among them, the server can be an independent server or a server cluster.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] See also Figure 1 , Figure 1 This is a schematic flow chart of a semantic segmentation method based on a SAM model provided by the first embodiment of the present application. The semantic segmentation method based on the SAM model can be applied to a server, and is used to generate a semantic label image by using the SAM model, which not only provides a category label for each pixel in the image, but also achieves accurate classification of each pixel in the image, and provides high-resolution semantic information for image analysis. By automatically determining the image segmentation mask and semantic features, the need for manual feature engineering is reduced, and the accuracy of the semantic segmentation dataset is improved.

[0029] like Figure 1 As shown, the semantic segmentation method based on the SAM model specifically includes steps S10 to S30.

[0030] Step S10: generating at least one set of semantically labeled images based on a preset semantic segmentation model and an initial image;

[0031] Step S20, determining at least one image segmentation mask of the initial image by using a preset image segmentation model SAM model and the initial image;

[0032] Step S30: determining target semantic features of the initial image based on the image segmentation masks and the semantic label images.

[0033] Specifically, before the emergence of the SAM segmentation model, there were already many semantic segmentation models that could be applied to various semantic segmentation datasets, but these semantic segmentation models were usually limited by the model size and data volume. Although they could provide classification masks for each category of the image, the classification results were not accurate enough, and the edges of the object classification were blurred. If the mask generated directly by the SAM model does not have category annotations, the semantic segmentation model trained on the specified dataset is limited by the amount of data and model size, resulting in inaccurate predictions. Therefore, this embodiment proposes to solve the above limitations by combining the fine image segmentation masks of SAM and SA-1B with the rich semantic annotations provided by other semantic segmentation models, making up for the shortcomings of these semantic segmentation models and bringing more accurate edges.

[0034] In one embodiment, the image input is first passed into the semantic segmentation model and the SAM branch respectively, and then passed into the semantic voting module together for fusion to obtain the final result. This embodiment uses the powerful generalization ability and image segmentation ability of SAM to improve the performance of the original model. By fusing the semantic voting module with the segmentation model results, a result that is both accurate and has category labels can be obtained, and better results can be obtained without retraining the semantic segmentation model. At the same time, the model has wide versatility, and the semantic segmentation model is not fixed. Any segmentation model can be selected to be fused with SAM.

[0035] The SAM model uses an attention mechanism to focus on key areas in an image, which helps the model identify and segment objects more accurately. The SAM model captures multi-scale information from coarse-grained to fine-grained by combining feature maps at different levels, which helps to better understand the image content. The SAM model is usually designed as an end-to-end architecture, which means that the entire process from the input image to the final segmentation result can be trained through a back-propagation algorithm. The SAM model is able to adapt to different images and segmentation tasks and optimize its performance by learning patterns in the data. Many SAM models are designed with real-time performance in mind, enabling them to process images quickly and are suitable for application scenarios that require fast response.

[0036] The characteristics of the SAM model include the following:

[0037] Zero-shot generalization: SAM has learned the general concept of objects and is able to perform zero-shot generalization to unfamiliar objects and images without additional training;

[0038] Multiple prompts: support interactive annotation, such as obtaining the segmented area by drawing a frame or clicking the mouse, and segmenting all objects in the image with one click;

[0039] Flexible integration: SAM prompts can be flexibly integrated with other systems, such as AR / VR interaction and text prompts;

[0040] Powerful dataset and training: More than 1 billion masks have been trained on 11 million images, achieving powerful image segmentation capabilities;

[0041] Interactive real-time segmentation: The SAM model can realize interactive real-time image segmentation, improving the efficiency and accuracy of image segmentation.

[0042] This embodiment discloses a semantic segmentation method, apparatus, device and storage medium based on a SAM model. The semantic segmentation method based on the SAM model includes generating at least one set of semantic label images based on a preset semantic segmentation model and an initial image; determining at least one image segmentation mask of the initial image through a preset image segmentation model SAM model and the initial image; and determining the target semantic features of the initial image based on each of the image segmentation masks and each of the semantic label images. In the above manner, the semantic label image generated by the present application using the SAM model not only provides a category label for each pixel in the image, but also realizes accurate classification of each pixel in the image, and provides high-resolution semantic information for image analysis. By automatically determining the image segmentation mask and semantic features, the need for manual feature engineering is reduced, and the accuracy of the semantic segmentation dataset is improved.

[0043] See also Figure 2 , Figure 2 This is a schematic flow chart of a semantic segmentation method based on a SAM model provided by the second embodiment of the present application. The semantic segmentation method based on the SAM model can be applied to a server to provide richer information by using the SAM model and fusion features, which helps the system to understand the image content more deeply, and improve the accuracy of segmentation by considering the relationship and importance between pixels. The calculation of weight scores can be adjusted according to different application requirements to optimize specific types of image analysis and improve the accuracy of semantic segmentation data sets.

[0044] based on Figure 1 The embodiment shown, this embodiment Figure 2 As shown, step S30 includes step S301 to step S302.

[0045] Step S301, fusing each of the image segmentation masks and each of the semantic label images to generate an image semantic fusion feature;

[0046] Step S302: Calculate the weight score of each of the image semantic fusion features, and determine the target semantic feature according to each of the weight scores.

[0047] In one embodiment, this embodiment can be implemented by the following steps:

[0048] First, we need to obtain image segmentation masks from the SAM model, which can identify the boundaries of different regions or objects in the image; at the same time, we need to obtain semantic label images from the same model, which provide a category label for each pixel, indicating the semantic category it belongs to.

[0049] The segmentation mask is combined with the semantically labeled image to create a fused feature map where each pixel contains not only category information but also spatial information, such as the location and shape of the object. For each pixel in the fused feature map, a weight score is calculated. This can be achieved in a variety of ways, for example, using the pixel's semantic category confidence, location information, and relationship to other objects in the image. Based on the calculated weight scores, the most important semantic features in the image are determined. These features may represent key objects or regions in the image and are critical to understanding the image content.

[0050] The present embodiment discloses a semantic segmentation method, apparatus, device and storage medium based on a SAM model. The semantic segmentation method based on the SAM model includes generating at least one set of semantic label images based on a preset semantic segmentation model and an initial image; determining at least one image segmentation mask of the initial image by using a preset image segmentation model SAM model and the initial image; fusing each of the image segmentation masks and each of the semantic label images to generate an image semantic fusion feature; calculating a weight score of each of the image semantic fusion features, and determining the target semantic feature according to each of the weight scores. In the above manner, the present application provides richer information by using the SAM model, which helps the system to understand the image content more deeply, and can improve the accuracy of segmentation by considering the relationship and importance between pixels. The calculation of the weight score can be adjusted according to different application requirements to optimize specific types of image analysis and improve the accuracy of the semantic segmentation data set.

[0051] In a specific embodiment, step S302 includes:

[0052] Evaluate all pixel points of each semantic label image and each image segmentation mask based on a preset voting rule to determine a weight score between pixel points of each semantic label image and pixel points of each image segmentation mask;

[0053] The image semantic fusion feature is edge segmented based on the weight score to determine the target semantic feature.

[0054] In a specific embodiment, a set of voting rules is defined to evaluate the consistency and importance of each pixel in the semantic label image and the image segmentation mask. For each pixel in the image, an evaluation is performed according to the preset voting rules to determine its weight score in the semantic label image and the segmentation mask. The weight score can be based on a variety of factors, such as the category confidence, spatial consistency, neighborhood information, etc. of the pixel. Using the calculated weight score, the image semantic fusion feature is edge segmented to determine the boundary of the target semantic feature. Based on the results of the edge segmentation, the target semantic features in the image are identified and determined, which represent key objects or areas in the image.

[0055] The present embodiment discloses a semantic segmentation method, device, equipment and storage medium based on a SAM model. The semantic segmentation method based on the SAM model includes generating at least one set of semantic label images based on a preset semantic segmentation model and an initial image; determining at least one image segmentation mask of the initial image by using a preset image segmentation model SAM model and the initial image; fusing each of the image segmentation masks and each of the semantic label images to generate an image semantic fusion feature; evaluating all pixels of each of the semantic label images and each of the image segmentation masks based on a preset voting rule to determine a weight score between the pixels of each of the semantic label images and the pixels of each of the image segmentation masks; performing edge segmentation on the image semantic fusion feature based on the weight score to determine the target semantic feature. In the above manner, the present application can more accurately identify the boundary of an object and improve the accuracy of segmentation by applying the voting rule. The calculation of the weight score helps to highlight important feature points and enhance the recognition of key semantic features. The voting mechanism can reduce the influence of noise and outliers and improve the robustness of the model. By considering the weights of pixels at different scales, multi-scale information can be better fused. The voting rules can be customized according to different application scenarios to adapt to different image analysis needs, thereby improving the accuracy of semantic segmentation datasets.

[0056] See also Figure 3 , Figure 3This is a schematic flow chart of a semantic segmentation method based on the SAM model provided in the third embodiment of the present application. The semantic segmentation method based on the SAM model can be applied to a server, and is used to quickly process images through a lightweight mask decoder, and is suitable for application scenarios that require real-time or near real-time feedback. Combining images and semantic embedding vectors can provide more accurate segmentation results, and the parameters of the encoder and decoder can be adjusted according to different application requirements, thereby improving the accuracy of the semantic segmentation data set.

[0057] based on Figure 1 The embodiment shown, this embodiment Figure 3 As shown, step S20 includes steps S201 to S203.

[0058] Step S201, generating an image embedding vector through the image encoder and the initial image;

[0059] Step S202, converting the semantic label image corresponding to the initial image into a semantic embedding vector through the prompt encoder;

[0060] Step S203: Determine, by the lightweight mask decoder, the at least one image segmentation mask according to the image embedding vector and the semantic embedding vector.

[0061] In one embodiment, the image encoder is part of a deep learning model that converts an input initial image into a feature embedding vector. This vector captures the high-level features and semantic information of the image. The initial image is processed by the image encoder to generate an image embedding vector, which is a highly compressed representation of the original image and contains the key visual features of the image. The hint encoder receives semantically labeled images as input and converts these images into semantic embedding vectors. These vectors contain feature information related to the semantic labels. The semantically labeled images are processed by the hint encoder to generate an embedding vector corresponding to the semantic category of each pixel in the image. The mask decoder is a model component that generates an image segmentation mask based on the image embedding vector and the semantic embedding vector. The lightweight design means that it can generate results quickly while maintaining efficient performance. The mask decoder combines the information of the image embedding vector and the semantic embedding vector to determine the segmentation mask for each pixel, which represents the confidence that the pixel belongs to a specific semantic category.

[0062] This embodiment discloses a semantic segmentation method, apparatus, device and storage medium based on the SAM model. The semantic segmentation method based on the SAM model includes generating at least one group of semantic label images based on a preset semantic segmentation model and an initial image; generating an image embedding vector through the image encoder and the initial image; converting the semantic label image corresponding to the initial image into a semantic embedding vector through the prompt encoder; determining at least one image segmentation mask according to the image embedding vector and the semantic embedding vector through the lightweight mask decoder; and determining the target semantic features of the initial image based on each of the image segmentation masks and each of the semantic label images. In the above manner, the present application can quickly process images through a lightweight mask decoder, which is suitable for application scenarios that require real-time or near real-time feedback. Combining images and semantic embedding vectors can provide more accurate segmentation results, and the parameters of the encoder and decoder can be adjusted according to different application requirements, thereby improving the accuracy of the semantic segmentation data set.

[0063] based on Figure 1 In the embodiment shown, in this embodiment, step S10 includes:

[0064] At least one group of the semantically labeled images matching the initial image is determined through a preset mapping relationship between images and labels in the preset semantic segmentation model and the initial image.

[0065] In a more preferred embodiment, before determining at least one set of the semantically labeled images matching the initial image through the preset mapping relationship between the image and the label in the preset semantic segmentation model and the initial image, the method includes:

[0066] Establishing a semantically labeled image sample library, wherein the semantically labeled image sample library includes image samples and label samples;

[0067] The image features of each of the image samples are extracted, and the preset mapping relationship between each of the image features and each of the label samples is established.

[0068] In one embodiment, a representative and diverse set of image samples is collected, which should cover all semantic categories that need to be identified. Accurate labels are provided for each image sample, which define the semantic category of the object or scene in the image. The collected image samples and corresponding labeled samples are organized into a structured database or dataset for subsequent processing and analysis. Features are extracted from the image samples using image processing techniques or deep learning models. These features may include color, texture, shape, position, etc. A mapping mechanism is determined to associate the extracted image features with the labeled samples. This usually involves encoding feature vectors and training classification models.

[0069] The present embodiment discloses a semantic segmentation method, apparatus, device and storage medium based on a SAM model, wherein the semantic segmentation method based on the SAM model includes establishing a semantic label image sample library, wherein the semantic label image sample library includes image samples and label samples; extracting image features of each of the image samples, and establishing the preset mapping relationship between each of the image features and each of the label samples; determining at least one group of the semantic label images matching the initial image through the preset mapping relationship between the image and the label in the preset semantic segmentation model and the initial image; determining at least one image segmentation mask of the initial image through the preset image segmentation model SAM model and the initial image; determining the target semantic features of the initial image based on each of the image segmentation masks and each of the semantic label images. In the above manner, the present application can improve the accuracy of image recognition through carefully selected features and optimized mapping relationships. By using a diverse sample library, the generalization ability of the model to new data can be improved. The establishment of a sample library and a mapping relationship provides convenience for adding new categories and features in the future, thereby improving the accuracy of the semantic segmentation data set.

[0070] Based on any of the above embodiments, in this embodiment, before step S10, the following steps are included:

[0071] Acquire training set data and verification set data, wherein the training set data includes training images and training labels, and the verification set data includes verification images and verification labels;

[0072] The initial SAM model is trained through each of the training images and each of the training labels, and the initial SAM model is iteratively verified through each of the verification images and each of the verification labels to determine the preset SAM model.

[0073] In one embodiment, sufficient training images and corresponding training labels are collected. The training labels should accurately mark the semantic categories of each object or region in the image. The training images are preprocessed, such as resizing and normalizing, to adapt to the input requirements of the SAM model.

[0074] Split a portion of the data from the training set as a validation set, or collect a separate set of validation images and validation labels. Use the training images and training labels to train the initial SAM model. During training, the model learns how to map input images to the correct semantic labels. Use the validation images and validation labels to evaluate the SAM model in training. This helps monitor the performance of the model, avoid overfitting, and perform hyperparameter tuning.

[0075] The performance of the model on the validation set is evaluated through various indicators (such as accuracy, precision, recall, F1 score, etc.). According to the performance feedback on the validation set, the model structure or training process is adjusted, such as changing the network layer, adjusting the learning rate, using regularization techniques, etc. After a series of iterative training and verification, a SAM model with the best performance is determined as the preset model.

[0076] The present embodiment discloses a semantic segmentation method, apparatus, device and storage medium based on a SAM model, wherein the semantic segmentation method based on the SAM model includes obtaining training set data and verification set data, wherein the training set data includes training images and training labels, and the verification set data includes verification images and verification labels; the initial SAM model is trained by each of the training images and each of the training labels, and the initial SAM model is iteratively verified by each of the verification images and each of the verification labels to determine the preset SAM model; at least one set of semantic label images is generated based on the preset semantic segmentation model and the initial image; at least one image segmentation mask of the initial image is determined by the preset image segmentation model SAM model and the initial image; based on each of the image segmentation masks and each of the semantic label images, the target semantic features of the initial image are determined. In the above manner, the present application can quickly iterate the model through an automated training and verification process, the application of the attention mechanism helps to improve the model's recognition accuracy of important information in the image, and the use of the verification set data can ensure that the model will not overfit, which helps to improve the generalization ability of the model and improve the accuracy of the semantic segmentation data set.

[0077] See also Figure 4 , Figure 4 The embodiment of the present application provides a schematic block diagram of a semantic segmentation device based on a SAM model, wherein the semantic segmentation device based on a SAM model is used to execute the aforementioned semantic segmentation method based on a SAM model. The semantic segmentation device based on a SAM model can be configured on a server.

[0078] like Figure 4 As shown, the semantic segmentation device 400 based on the SAM model includes:

[0079] A semantic label image generation module 410, configured to generate at least one set of semantic label images based on a preset semantic segmentation model and an initial image;

[0080] An image segmentation mask generating module 420, configured to determine at least one image segmentation mask of the initial image by using a preset image segmentation model SAM model and the initial image;

[0081] The target semantic feature determination module 430 is used to determine the target semantic feature of the initial image based on each of the image segmentation masks and each of the semantic label images.

[0082] Furthermore, the target semantic feature determination module 430 includes:

[0083] An image semantic fusion feature generating unit, used for fusing each of the image segmentation masks and each of the semantic label images to generate an image semantic fusion feature;

[0084] The target semantic feature determination unit is used to calculate the weight score of each of the image semantic fusion features and determine the target semantic feature according to each of the weight scores.

[0085] Furthermore, the target semantic feature determination unit includes:

[0086] A weight score determination subunit, used to evaluate all pixel points of each of the semantic label images and each of the image segmentation masks based on a preset voting rule, and determine a weight score between the pixel points of each of the semantic label images and the pixel points of each of the image segmentation masks;

[0087] The target semantic feature determination subunit is used to perform edge segmentation on the image semantic fusion feature based on the weight score to determine the target semantic feature.

[0088] Furthermore, the image segmentation mask generation module 420 includes:

[0089] An image embedding vector generating unit, configured to generate an image embedding vector through the image encoder and the initial image;

[0090] A semantic embedding vector conversion unit, configured to convert the semantic label image corresponding to the initial image into a semantic embedding vector through the hint encoder;

[0091] An image segmentation mask generating unit is used to determine the at least one image segmentation mask according to the image embedding vector and the semantic embedding vector through the lightweight mask decoder.

[0092] Furthermore, the semantic label image generation module 410 includes:

[0093] The semantic label image generating unit is used to determine at least one group of the semantic label images matching the initial image through the preset mapping relationship between the image and the label in the preset semantic segmentation model and the initial image.

[0094] Furthermore, the semantic label image generating unit comprises:

[0095] A semantic label image sample library establishing subunit, used to establish a semantic label image sample library, wherein the semantic label image sample library includes image samples and label samples;

[0096] The preset mapping relationship establishing subunit is used to extract the image features of each of the image samples and establish the preset mapping relationship between each of the image features and each of the label samples.

[0097] Furthermore, the semantic segmentation device 400 based on the SAM model further includes:

[0098] A training data acquisition module, used to acquire training set data and verification set data, wherein the training set data includes training images and training labels, and the verification set data includes verification images and verification labels;

[0099] The SAM model training module is used to train the initial SAM model through each of the training images and each of the training labels, and to iteratively verify the initial SAM model through each of the verification images and each of the verification labels to determine the preset SAM model.

[0100] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0101] The above-mentioned device can be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.

[0102] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0103] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0104] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any semantic segmentation method based on the SAM model.

[0105] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0106] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any semantic segmentation method based on the SAM model.

[0107] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0109] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0110] Based on a preset semantic segmentation model and an initial image, generating at least one set of semantically labeled images;

[0111] Determine at least one image segmentation mask of the initial image by using a preset image segmentation model SAM model and the initial image;

[0112] Based on each of the image segmentation masks and each of the semantic label images, a target semantic feature of the initial image is determined.

[0113] In one embodiment, based on each of the image segmentation masks and each of the semantic label images, a target semantic feature of the initial image is determined to achieve:

[0114] Fusing each of the image segmentation masks and each of the semantic label images to generate an image semantic fusion feature;

[0115] The weight score of each of the image semantic fusion features is calculated, and the target semantic feature is determined according to each of the weight scores.

[0116] In one embodiment, the weight score of each of the image semantic fusion features is calculated, and the target semantic feature is determined according to each of the weight scores, so as to achieve:

[0117] Evaluate all pixel points of each semantic label image and each image segmentation mask based on a preset voting rule to determine a weight score between pixel points of each semantic label image and pixel points of each image segmentation mask;

[0118] The image semantic fusion feature is edge segmented based on the weight score to determine the target semantic feature.

[0119] In one embodiment, at least one image segmentation mask of the initial image is determined by presetting an image segmentation model SAM model and the initial image, so as to achieve:

[0120] Generate an image embedding vector by using the image encoder and the initial image;

[0121] Converting the semantic label image corresponding to the initial image into a semantic embedding vector by the hint encoder;

[0122] The at least one image segmentation mask is determined by the lightweight mask decoder according to the image embedding vector and the semantic embedding vector.

[0123] In one embodiment, based on a preset semantic segmentation model and an initial image, at least one set of semantically labeled images is generated to achieve:

[0124] At least one group of the semantically labeled images matching the initial image is determined through a preset mapping relationship between images and labels in the preset semantic segmentation model and the initial image.

[0125] In one embodiment, before determining at least one set of the semantically labeled images matching the initial image through the preset mapping relationship between the image and the label in the preset semantic segmentation model and the initial image, it is used to implement:

[0126] Establishing a semantically labeled image sample library, wherein the semantically labeled image sample library includes image samples and label samples;

[0127] The image features of each of the image samples are extracted, and the preset mapping relationship between each of the image features and each of the label samples is established.

[0128] In one embodiment, by presetting the image segmentation model SAM model and the initial image, before determining at least one image segmentation mask of the initial image, it is used to implement:

[0129] Acquire training set data and verification set data, wherein the training set data includes training images and training labels, and the verification set data includes verification images and verification labels;

[0130] The initial SAM model is trained through each of the training images and each of the training labels, and the initial SAM model is iteratively verified through each of the verification images and each of the verification labels to determine the preset SAM model.

[0131] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the semantic segmentation methods based on the SAM model provided in the embodiments of the present application.

[0132] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Medi a Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card), etc., equipped on the computer device.

[0133] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A semantic segmentation method based on the SAM model, characterized in that: include: Based on a preset semantic segmentation model and an initial image, generating at least one set of semantically labeled images; Determine at least one image segmentation mask of the initial image by using a preset image segmentation model SAM model and the initial image; Based on each of the image segmentation masks and each of the semantic label images, a target semantic feature of the initial image is determined.

2. The semantic segmentation method based on the SAM model according to claim 1, characterized in that: The step of determining the target semantic features of the initial image based on each of the image segmentation masks and each of the semantic label images comprises: Fusing each of the image segmentation masks and each of the semantic label images to generate an image semantic fusion feature; The weight score of each of the image semantic fusion features is calculated, and the target semantic feature is determined according to each of the weight scores.

3. The semantic segmentation method based on the SAM model according to claim 2, characterized in that: The calculating the weight score of each of the image semantic fusion features and determining the target semantic feature according to each of the weight scores includes: Evaluate all pixel points of each semantic label image and each image segmentation mask based on a preset voting rule to determine a weight score between pixel points of each semantic label image and pixel points of each image segmentation mask; The image semantic fusion feature is edge segmented based on the weight score to determine the target semantic feature.

4. The semantic segmentation method based on the SAM model according to claim 1, characterized in that: The preset SAM model includes an image encoder, a hint encoder and a lightweight mask decoder, and determining at least one image segmentation mask of the initial image through the preset image segmentation model SAM model and the initial image includes: Generate an image embedding vector by using the image encoder and the initial image; Converting the semantic label image corresponding to the initial image into a semantic embedding vector by the hint encoder; The at least one image segmentation mask is determined by the lightweight mask decoder according to the image embedding vector and the semantic embedding vector.

5. The semantic segmentation method based on the SAM model according to claim 1, characterized in that: The step of generating at least one set of semantically labeled images based on a preset semantic segmentation model and an initial image includes: At least one group of the semantically labeled images matching the initial image is determined through a preset mapping relationship between images and labels in the preset semantic segmentation model and the initial image.

6. The semantic segmentation method based on the SAM model according to claim 5, characterized in that: Before determining at least one group of the semantically labeled images matching the initial image through the preset mapping relationship between the image and the label in the preset semantic segmentation model and the initial image, the method includes: Establishing a semantically labeled image sample library, wherein the semantically labeled image sample library includes image samples and label samples; The image features of each of the image samples are extracted, and the preset mapping relationship between each of the image features and each of the label samples is established.

7. The semantic segmentation method based on the SAM model according to any one of claims 1 to 6, characterized in that: Before determining at least one image segmentation mask of the initial image by using the preset image segmentation model SAM model and the initial image, the method includes: Acquire training set data and verification set data, wherein the training set data includes training images and training labels, and the verification set data includes verification images and verification labels; The initial SAM model is trained through each of the training images and each of the training labels, and the initial SAM model is iteratively verified through each of the verification images and each of the verification labels to determine the preset SAM model.

8. A semantic segmentation device based on the SAM model, characterized in that: include: A semantic label image generation module, used to generate at least one set of semantic label images based on a preset semantic segmentation model and an initial image; An image segmentation mask generating module, used to determine at least one image segmentation mask of the initial image by using a preset image segmentation model SAM model and the initial image; The target semantic feature determination module is used to determine the target semantic feature of the initial image based on each of the image segmentation masks and each of the semantic label images.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the semantic segmentation method based on the SAM model as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the semantic segmentation method based on the SAM model as described in any one of claims 1 to 7.

Images