Method for training a prompt-free image semantic segmentation model and application thereof

Abstract

This invention discloses a training method and application for a prompt-free image semantic segmentation model. The training method includes: acquiring a training dataset comprising multiple image samples, each image sample including category labeling information; extracting and fusing general visual features and general scene-specific features from the multiple image samples to obtain image fusion features; generating category prompting information for the corresponding image samples based on the image fusion features; and adjusting the parameters of the image semantic segmentation model based on a preset loss function until the training objective is achieved, wherein the preset loss function includes an error loss term for the category prompting information and the category labeling information. Therefore, the image semantic segmentation model trained using this invention can automatically and accurately perform semantic segmentation of images without external input prompting information, effectively assisting professionals in related fields in target recognition and analysis.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a training method and application of a prompt-free image semantic segmentation model. Background Technology

[0002] Image semantic segmentation is a core technology in computer vision. It divides images into independent regions representing different targets and structural types, providing crucial support for various tasks such as target detection, scene analysis, and intelligent monitoring. With the development of image acquisition technology, the amount of image data accumulated in various fields has surged, making the demand for automated and high-precision segmentation algorithms increasingly urgent.

[0003] However, existing technologies require manual prompts (such as dot hints and box hints) to generate target region masks in image semantic segmentation tasks. This relies on professional personnel and is inefficient when faced with a large number of image semantic segmentation tasks. The results are also affected by human factors and cannot meet the needs of automated image analysis.

[0004] Therefore, to address the aforementioned technical issues, it is necessary to provide a training method and application for a prompt-free image semantic segmentation model. Summary of the Invention

[0005] The purpose of this invention is to provide a training method and application for a prompt-free image semantic segmentation model, which can solve the problem that existing technologies rely on externally provided prompts, resulting in low efficiency in image semantic segmentation.

[0006] To achieve the above objectives, a specific embodiment of the present invention provides a training method for a prompt-free image semantic segmentation model, the method comprising:

[0007] Obtain a training dataset, wherein the training dataset includes multiple image samples, and the image samples include category annotation information;

[0008] The general visual features and general scene-specific features of the multiple image samples are extracted and fused to obtain image fusion features;

[0009] Based on the image fusion features, category prompt information is generated for the corresponding image samples;

[0010] Based on a preset loss function, the parameters of the image semantic segmentation model are adjusted until the training objective is achieved. The preset loss function includes an error loss term for the category prompt information and the category annotation information.

[0011] In one or more embodiments of the present invention, the image semantic segmentation model includes a classification module, which is used to generate category prompt information for the corresponding image samples based on the image fusion features. The classification module includes a multi-layer network constructed based on a trainable activation function matrix, and each layer of the network includes residual connections and convolutional alignment units.

[0012] In one or more embodiments of the present invention, the input of each layer of the network in the classification module is directly superimposed to the feature output of the corresponding layer through the residual connection, and after being superimposed with the feature output of the corresponding layer, it is processed by the convolutional alignment unit and used as the final output of the corresponding layer.

[0013] In one or more embodiments of the present invention, the method includes:

[0014] The general visual features and the general-specific features are fused using a feature stitching method; and / or,

[0015] Based on the image fusion features, category hint information is generated for the corresponding image samples, specifically including:

[0016] The image fusion features are reduced in dimensionality and integrated to obtain low-dimensional image fusion features;

[0017] Based on the low-dimensional image fusion features, category hint information is generated for the corresponding image samples.

[0018] In one or more embodiments of the present invention, the category labeling information includes a true category mask, and the category hint information includes a predicted category mask;

[0019] The preset loss function includes at least one of a first loss function sub-item, a second loss function sub-item, and a third loss function sub-item. The first loss function sub-item represents the degree of overlap between the true class mask and the predicted class mask. The second loss function sub-item represents the prediction error of the target image sample. The third loss function sub-item is used to represent the error between the degree of overlap between the true mask and the degree of overlap between the predicted mask.

[0020] The target image samples include image samples whose classification confidence is lower than a preset threshold output by the image semantic segmentation model, and the degree of overlap of the real mask is calculated based on the degree of overlap between the real class mask and the predicted class mask.

[0021] A specific embodiment of the present invention also provides a prompt-free image semantic segmentation method, the method comprising:

[0022] Obtain the image to be semantically segmented;

[0023] The image to be semantically segmented is processed based on a prompt-free image semantic segmentation model to obtain a semantic segmentation mask;

[0024] The image semantic segmentation model is trained using the training method described above.

[0025] A specific embodiment of the present invention also provides a training apparatus for a prompt-free image semantic segmentation model, comprising:

[0026] An acquisition module is used to acquire a training dataset, wherein the training dataset includes multiple image samples, and the image samples include category labeling information;

[0027] The extraction and fusion module is used to extract the general visual features and general scene-specific features of the multiple image samples respectively and fuse them to obtain image fusion features;

[0028] The prompting module is used to generate category prompting information for the corresponding image samples based on the image fusion features;

[0029] The training module is used to adjust the parameters of the image semantic segmentation model based on a preset loss function until the training objective is achieved. The preset loss function includes an error loss term for the category prompt information and the category annotation information.

[0030] A specific embodiment of the present invention also provides a prompt-free image semantic segmentation device, comprising:

[0031] The acquisition module is used to acquire the image to be semantically segmented;

[0032] The processing module is used to process the image to be semantically segmented based on the prompt-free image semantic segmentation model to obtain a semantic segmentation mask;

[0033] The image semantic segmentation model is trained using the training method described above.

[0034] A specific embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a training method for a prompt-free image semantic segmentation model as described above or a prompt-free image semantic segmentation method as described above.

[0035] A specific embodiment of the present invention also provides a machine-readable storage medium storing executable instructions, which, when executed, cause the machine to perform a training method for a prompt-free image semantic segmentation model as described above, or a prompt-free image semantic segmentation method as described above.

[0036] Compared with existing technologies, the training method of the prompt-free image semantic segmentation model of the present invention extracts and fuses general visual features and general scene-specific features from multiple image samples, and generates category prompts for the corresponding image samples based on the image fusion features. On the one hand, feature fusion can reduce computational complexity while preserving the original feature information, balancing model performance and operating efficiency; on the other hand, generating category prompts based on fusion features enables automatic and high-precision semantic segmentation of images without the need for manual input of prompts, improving the efficiency and objectivity of image analysis. Furthermore, by setting a preset loss function to adjust the parameters of the image semantic segmentation model, and the preset loss function including error loss terms for category prompts and category annotations, it can effectively solve problems such as class imbalance and unclear segmentation boundaries in image samples, further optimizing the accuracy of the image semantic segmentation mask. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This diagram illustrates the training method and implementation environment of the prompt-free image semantic segmentation model in one embodiment of the present invention.

[0039] Figure 2 This is a flowchart of a training method for a prompt-free image semantic segmentation model according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the framework of a prompt-free image semantic segmentation model in one embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the framework of the classification module in one embodiment of the present invention;

[0042] Figure 5 This is a flowchart of a prompt-free image semantic segmentation method in one embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the structure of a training device for a prompt-free image semantic segmentation model in one embodiment of the present invention;

[0044] Figure 7 This is a schematic diagram of the structure of a prompt-free image semantic segmentation device in one embodiment of the present invention;

[0045] Figure 8This is a hardware structure diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0047] Image semantic segmentation, a core technology in computer vision, supports tasks such as object detection and scene analysis by dividing images into independent regions. With the surge in image data across various fields, the demand for automated, high-precision segmentation algorithms is becoming increasingly urgent.

[0048] Among them, SAM and its upgraded version SAM2 model have made breakthrough progress in the field of image segmentation. Compared with SAM, SAM2 has significantly improved in segmentation accuracy, generalization performance and video segmentation ability, and has been initially applied to general image segmentation scenarios.

[0049] However, existing SAM / SAM2-based segmentation methods require frequent manual prompts (such as dot hints and bounding box hints) to generate target region masks. This reliance on professional personnel makes them inefficient and susceptible to human error when faced with a large volume of image semantic segmentation tasks, failing to meet the demands of automated image analysis. Furthermore, the original design of SAM2 was not optimized for the diverse scenarios of general image semantic segmentation tasks, lacking prior knowledge across different domains and making it difficult to accurately identify complex target morphologies and structural features in images.

[0050] The training method and application of the prompt-free image semantic segmentation model of the present invention can effectively solve the above-mentioned technical problems. It does not rely on external prompting information, thus improving the accuracy and objectivity of image semantic segmentation masks. At the same time, it improves the segmentation accuracy for diverse scenes, which can meet the needs of automated image analysis.

[0051] Reference Figure 1The diagram illustrates an implementation environment provided by an exemplary embodiment of the present invention. This implementation environment includes a terminal and a server. The terminal and server communicate via a communication network, which can optionally be a wired network or a wireless network, and can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN). The prompt-free image semantic segmentation method disclosed in this invention can be executed by the server, and correspondingly, the prompt-free image semantic segmentation model can also be deployed on the server. Users can input an image to be semantically segmented via the terminal, and the server can obtain the semantic segmentation mask corresponding to the input image by running the prompt-free image semantic segmentation model.

[0052] Alternatively, the terminal and server can work together to run the promptless image semantic segmentation method provided in the embodiments of this application to obtain the semantic segmentation mask corresponding to the input image.

[0053] In a system architecture where the terminal can provide matching computing power, the prompt-free image semantic segmentation method disclosed in this application can also be directly executed by the terminal, and correspondingly, the prompt-free image semantic segmentation model can also be deployed on the terminal.

[0054] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0055] Reference Figure 2 and Figure 3 This invention discloses a training method for a prompt-free image semantic segmentation model, comprising:

[0056] S101. Obtain the training dataset, which includes multiple image samples and category labeling information for each image sample.

[0057] Specifically, image samples to be segmented can be obtained first, and these samples can be selected from multiple publicly available image datasets. Next, the image samples are standardized in size and pixel values ​​are normalized to obtain preprocessed image samples. Finally, a training dataset is obtained based on the preprocessed image samples.

[0058] Furthermore, a subset of image samples can be extracted from the training dataset as a validation set for subsequent parameter tuning of the prompt-free image semantic segmentation model. This embodiment uses 10% as the validation set as an example; in other embodiments, other proportions of data can be extracted as the validation set, and this application does not impose any limitations on this.

[0059] S102. Extract general visual features and general scene-specific features from multiple image samples and fuse them to obtain image fusion features. The general visual features are not visual features specific to a particular task or scene, but rather the model's understanding of basic visual attributes such as object boundaries, textures, shapes, and contours in the image. The general scene-specific features are not just scene-specific features within a very narrow domain, but cover various general scenarios, such as gesture recognition and multi-object detection, exhibiting broad domain adaptability.

[0060] Specifically, a first encoder can extract general visual features from image samples, while a second encoder can extract general scene-specific features from the image samples. In this embodiment, the first encoder can be the SAM2 encoder, and the second encoder can be a general-purpose pre-trained UNI encoder. The SAM2 encoder is an upgraded version of SAM, offering significant improvements in segmentation accuracy, generalization performance, and video segmentation capabilities. The UNI encoder is a self-supervised model pre-trained on over 100,000 images from various scenes, covering more than 20 major target types (including common and special targets in different scenes). Compared to the SAM2 encoder's general segmentation capability for "everything," the UNI encoder's features focus more on semantic categories. It can imbue image features with high-level semantic meaning, determining "what type of object it is (e.g., a car, a pedestrian, or a gesture)," providing a basis for generating accurate category prompts subsequently.

[0061] Because the original design of the SAM2 encoder was not optimized for the diverse scenarios of general image semantic segmentation tasks, it lacks prior knowledge in different domain scenarios, making it difficult to accurately identify complex target shapes and structural features in images. Therefore, the SAM2 encoder was adapted to general image semantic segmentation tasks, and by integrating the UNI encoder, prior knowledge of general scene domains was injected, solving the problem of SAM2's lack of professional knowledge of general scenes and significantly improving the model's segmentation accuracy and generalization ability.

[0062] In this embodiment, a SAM2 encoder (with some underlying weights frozen) and a UNI encoder (with pre-trained weights frozen) are loaded, and both process the input image in parallel. By freezing the parameters of the pre-trained SAM2 and UNI encoders, their feature extraction capabilities can be ensured to remain stable.

[0063] Furthermore, a feature concatenation method can be used to fuse general visual features with general-specific features. Specifically, the Concat method can be used to concatenate the output features of the SAM2 encoder (dimension C1×H×W) and the output features of the UNI encoder (dimension C2×H×W) in the channel dimension to obtain the fused feature (dimension (C1+C2)×H×W)).

[0064] S103. Based on image fusion features, generate category prompt information for the corresponding image samples.

[0065] Specifically, this includes: performing dimensionality reduction and feature integration on image fusion features to obtain low-dimensional image fusion features; and generating category hints for the corresponding image samples based on the low-dimensional image fusion features. The following provides a detailed description of the above methods.

[0066] Image fusion features can be reduced in dimensionality and integrated using an input dimension alignment module to obtain low-dimensional image fusion features. Therefore, a dual-encoder feature concatenation and fusion design with dimension alignment is adopted, which reduces computational complexity while preserving the original feature information, thus balancing model performance and running efficiency.

[0067] Specifically, the dimension alignment module can be constructed from a dimensionality reduction network consisting of two convolutional layers. The first layer has a kernel size of 3×3, a stride of 1, padding of 1, and an output dimension of C3, with ReLU activation. The second layer has a kernel size of 1×1 and an output dimension of C4 (which can be adapted to the input of the subsequent classification module). The above is merely an illustrative example of the dimension alignment module and is not intended to limit it.

[0068] Based on low-dimensional image fusion features, category hints are generated for the corresponding image samples.

[0069] Specifically, the image semantic segmentation model includes a classification module, which generates category prompts for corresponding image samples based on image fusion features. The classification module includes a multi-layer network built on a trainable activation function matrix, and each layer of the network includes residual connections and convolutional alignment units.

[0070] In this embodiment, the classification module can be a Residual Kolmogorov-Arnold Network (ResKAN) classification module. The ResKAN classification module can consist of k KAN layers, each with residual connections and convolutional alignment units.

[0071] In this embodiment, please refer to Figure 4The ResKAN classification module may include a 3-layer KAN network, where the activation function matrix of each KAN layer has a dimension of [input dimension × output dimension]. Each layer adds residual connections and 3×3 convolutional alignment units, and the output dimension is the number of classes × the cue token dimension (e.g., for an 8-class segmentation task, the output is an 8×256 dimension cue token). In other embodiments, the number of KAN layers can also be adjusted, and this application does not limit this.

[0072] In contrast, some image semantic segmentation models use Multilayer Perceptron (MLP) as the classification prompt module, which suffers from low parameter utilization efficiency, poor interpretability, and insufficient integration of large-scale general image pre-training knowledge, resulting in poor segmentation accuracy. In this application, category prompts are generated for corresponding image samples based on image fusion features. Furthermore, the ResKAN classification module, combined with the high flexibility, high interpretability, and gradient stability of the residual Kolmogorov-Arnold network, replaces the MLP and manual prompting process, achieving prompt-free automatic segmentation, improving segmentation efficiency, and avoiding the uncertainties caused by human intervention.

[0073] Furthermore, the input of each layer in the classification module is directly superimposed on the feature output of the corresponding layer through residual connections. After being superimposed on the feature output of the corresponding layer, it is processed by the convolutional alignment unit to become the final output of the corresponding layer. Through residual connections, it can avoid deep networks from not learning anything. The input of each layer is directly superimposed on the feature output of the corresponding layer through residual connections, which is equivalent to opening a "shortcut" for information. Even if the KAN layer does not learn well, the original information can still be preserved.

[0074] S104. Based on a preset loss function, adjust the parameters of the image semantic segmentation model until the training objective is achieved. The preset loss function includes error loss terms for category hints and category annotations. The category annotations include the ground truth class mask, and the category hints include the predicted class mask. In this embodiment, the model parameters can be optimized through backpropagation.

[0075] The preset loss function includes at least one of a first loss function sub-term, a second loss function sub-term, and a third loss function sub-term. The first loss function sub-term represents the degree of overlap between the ground truth class mask and the predicted class mask; the second loss function sub-term represents the prediction error of the target image sample; and the third loss function sub-term represents the error between the degree of overlap between the ground truth mask and the degree of overlap between the predicted class mask. The target image sample includes image samples whose classification confidence output by the image semantic segmentation model is lower than a preset threshold; that is, the target image sample is an image sample in the training dataset that is difficult to classify. The degree of overlap between the ground truth mask and the predicted class mask is calculated.

[0076] In this embodiment, the first loss function can be the Dice loss function, the second loss function can be the Focal loss function, and the third loss function can be the MSE loss function. The Dice loss function, based on the Dice coefficient, evaluates the overlap between the ground truth class mask and the predicted class mask. The Focal loss function dynamically reduces the weight of easily classified samples during training, thus quickly focusing the attention on the target image samples (difficult-to-classify image samples). The MSE loss function calculates the mean squared error between the predicted and actual cross-union ratios (CUP). The CUP directly reflects the error between the overlap between the ground truth mask and the predicted mask.

[0077] The preset loss function can include any one of the three, any two of the three, or all of the three. Furthermore, the preset loss function can be weighted to adjust the proportion of different loss function sub-items.

[0078] In this embodiment, the preset loss function includes the above three sub-items, and the formula is as follows:

[0079]

[0080] in, and For weight hyperparameters. Denotes the Dice loss function. Represents the Focal loss function. This represents the mean squared error loss function.

[0081] In one embodiment of this invention, the prompt-free image semantic segmentation model uses AdamW as the optimizer during training, with a weight decay coefficient of 1e-5 and an initial learning rate of 1e-4, employing cosine annealing for learning rate scheduling. The batch size during training is set to 8, and the number of training epochs is set to 100. Training stops when the validation set IOU (Intersection over Union) shows no improvement for 10 consecutive epochs, and the optimal model weights are saved. Simultaneously, a regularization strategy is employed, specifically adding Dropout layers (dropout rate = 0.1) to the dimension alignment module and the ResKAN classification module to prevent overfitting. The above are merely examples of hyperparameters and strategies for the prompt-free image semantic segmentation model during training and are not intended to limit its scope.

[0082] Therefore, the model obtained by the training method of the prompt-free image semantic segmentation model of the present invention can be effectively applied to general image segmentation scenarios. It eliminates the need for manual input of prompts, improving the efficiency and objectivity of image analysis. Simultaneously, by fusing features and performing dimensionality reduction, it can reduce computational complexity while preserving original feature information, balancing model performance and operational efficiency. Finally, by setting a preset loss function, it can effectively solve problems such as class imbalance and unclear segmentation boundaries in image samples, further optimizing the accuracy of the image semantic segmentation mask and the IOU prediction precision.

[0083] The technical effects are explained below with specific experimental data.

[0084] The datasets used in the experiment are as follows, which are derived from multiple publicly available and commonly used image datasets.

[0085] Dataset 1: 4456 images of various scenes, including 6 target types, with a uniform image size of 256×256 pixels;

[0086] Dataset 2: 213 images, derived from 38 high-resolution images magnified 20x, with an image size of approximately 512×512 pixels;

[0087] Dataset 3: 165 target images of a specific scene, with a size of 768×768 pixels.

[0088]

[0089] Among them, the Dice similarity coefficient (DSC) and intersection-union ratio (IoU) are two core metrics for evaluating the overlap between the predicted and ground truth masks in semantic segmentation tasks. DSC is defined as twice the area of ​​the intersection between the predicted and ground truth regions divided by the sum of their areas, focusing on measuring the similarity between the two sets; IoU is defined as the area of ​​the intersection divided by the area of ​​the union, more intuitively reflecting the overlap ratio between the predicted and ground truth regions. Both can effectively evaluate the segmentation accuracy of the model.

[0090] Based on the experimental data above, the ablation experiments verify the advantages of the General-SAM2 model proposed in this invention. Specifically, compared with the MLP scheme and the ordinary SAM / SAM2 scheme, the DSC and IOU values ​​of this invention, through the ResKAN classification module, are superior on datasets 1 to 3, proving the effectiveness of combining the residual structure with the KAN architecture.

[0091] Therefore, the prompt-free image semantic segmentation model obtained by this training method outperforms existing baseline models in terms of DSC and IOU metrics on multiple publicly available image datasets. It can effectively assist professionals in related fields to perform target analysis and has broad application value.

[0092] Reference Figure 5 This invention discloses a prompt-free image semantic segmentation method in one embodiment, comprising:

[0093] S201. Obtain the image to be semantically segmented;

[0094] S202. Image processing for semantic segmentation based on the prompt-free image semantic segmentation model to obtain a semantic segmentation mask; wherein, the image semantic segmentation model is trained by the above training method.

[0095] In the runtime of the prompt-free image semantic segmentation model, the image to be segmented is input into the trained model, which then undergoes preprocessing, feature extraction, feature fusion and alignment, classification module prompt generation, and decoding output steps to obtain a semantic segmentation mask. This mask is then superimposed on the original image and output for review by relevant personnel. Furthermore, the prompt-free image semantic segmentation model also generates a semantic segmentation mask for the image to be semantically segmented based on low-dimensional image fusion features and category prompt information.

[0096] Therefore, the image semantic segmentation model trained based on the above-mentioned prompt-free image semantic segmentation model can perform semantic segmentation on images and has accurate semantic segmentation results.

[0097] Reference Figure 6 This invention discloses a training device for a prompt-free image semantic segmentation model in one embodiment, including an acquisition module 301, an extraction and fusion module 302, a prompting module 303, and a training module 304.

[0098] The acquisition module 301 is used to acquire the training dataset, which includes multiple image samples, and the image samples include category labeling information; the extraction and fusion module 302 is used to extract the general visual features and general scene-specific features of the multiple image samples respectively and fuse them to obtain image fusion features; the prompting module 303 is used to generate category prompting information for the corresponding image samples based on the image fusion features; the training module 304 is used to adjust the parameters of the image semantic segmentation model based on a preset loss function until the training objective is achieved, wherein the preset loss function includes an error loss term for the category prompting information and the category labeling information.

[0099] In one optional embodiment, the image semantic segmentation model includes a classification module, which is used to generate category prompt information for the corresponding image samples based on image fusion features. The classification module includes a multi-layer network constructed based on a trainable activation function matrix, and each layer of the network includes residual connections and convolutional alignment units.

[0100] In one optional embodiment, the input of each layer of the classification module is directly superimposed to the feature output of the corresponding layer through residual connections, and after being superimposed with the feature output of the corresponding layer, it is processed by the convolutional alignment unit to become the final output of the corresponding layer.

[0101] In one optional embodiment, the extraction and fusion module 302 uses a feature stitching method to fuse general visual features with general special features.

[0102] In one optional embodiment, the prompting module 303 generates category prompt information for the corresponding image sample based on image fusion features, specifically including: performing dimensionality reduction and feature integration on the image fusion features to obtain low-dimensional image fusion features; and generating category prompt information for the corresponding image sample based on the low-dimensional image fusion features.

[0103] In one optional embodiment, the category labeling information includes a ground truth class mask, and the category hint information includes a predicted class mask. The preset loss function includes at least one of a first loss function sub-item, a second loss function sub-item, and a third loss function sub-item. The first loss function sub-item represents the degree of overlap between the ground truth class mask and the predicted class mask; the second loss function sub-item represents the prediction error of the target image sample; and the third loss function sub-item represents the error between the degree of overlap between the ground truth mask and the degree of overlap between the predicted mask.

[0104] The target image samples include image samples whose classification confidence is lower than a preset threshold output by the image semantic segmentation model. The degree of overlap of the real mask is calculated based on the degree of overlap between the real class mask and the predicted class mask.

[0105] Reference Figure 7 This invention introduces a prompt-free image semantic segmentation device according to an embodiment of the present invention, including an acquisition module 401 and a processing module 402.

[0106] The acquisition module 401 is used to acquire the image to be semantically segmented; the processing module 402 is used to process the image to be semantically segmented based on the prompt-free image semantic segmentation model to obtain a semantic segmentation mask; wherein, the image semantic segmentation model is trained by the above-mentioned training method.

[0107] As referred above Figures 1 to 5 This specification describes a training method for a prompt-free image semantic segmentation model and a prompt-free image semantic segmentation method according to embodiments thereof. The details mentioned in the above description of the method embodiments also apply to the prompt-free image semantic segmentation apparatus of embodiments thereof. The above-described prompt-free image semantic segmentation apparatus can be implemented in hardware, software, or a combination of hardware and software.

[0108] Reference Figure 8This diagram illustrates a hardware structure of an electronic device according to an embodiment of this specification. The electronic device 50 may include at least one processor 51, a memory 52 (e.g., non-volatile memory), a main memory 53, and a communication interface 54, and the at least one processor 51, memory 52, main memory 53, and communication interface 54 are connected together via an internal bus 55. At least one processor 51 executes at least one computer-readable instruction stored or encoded in the memory 52.

[0109] It should be understood that the computer-executable instructions stored in memory 52, when executed, cause at least one processor 51 to perform the above-described combinations in the various embodiments of this specification. Figures 1 to 5 The description includes various operations and functions.

[0110] In the embodiments of this specification, electronic device 50 may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile electronic device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable electronic device, consumer electronic device, etc.

[0111] According to one embodiment, a program product, such as a machine-readable medium, is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the above-described combinations of the various embodiments of this specification. Figures 1-5 The various operations and functions described. Specifically, a system or apparatus equipped with a readable storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer or processor of the system or apparatus to read and execute the instructions stored in the readable storage medium.

[0112] In this case, the program code read from the readable medium itself can perform the functions of any of the above embodiments, and therefore the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.

[0113] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0114] Those skilled in the art will understand that the various embodiments disclosed above can be modified and varied without departing from the spirit of the invention. Therefore, the scope of protection of this specification should be defined by the appended claims.

[0115] It should be noted that not all steps and units in the above process and system structure diagrams are mandatory; some steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or they may be jointly implemented by certain components in multiple independent devices.

[0116] In the above embodiments, the hardware units or modules can be implemented mechanically or electrically. For example, a hardware unit, module, or processor may include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. The hardware unit or processor may also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical, dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.

[0117] The specific embodiments described above with reference to the accompanying drawings are exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" compared to other embodiments. Specific details are included to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0118] The foregoing description of this disclosure is provided to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles applicable herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

[0119] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A training method for a prompt-free image semantic segmentation model, characterized in that, The method includes: Obtain a training dataset, wherein the training dataset includes multiple image samples, and the image samples include category annotation information; The general visual features and general scene-specific features of the multiple image samples are extracted and fused to obtain image fusion features; Based on the image fusion features, category prompt information is generated for the corresponding image samples; Based on a preset loss function, the parameters of the image semantic segmentation model are adjusted until the training objective is achieved. The preset loss function includes an error loss term for the category prompt information and the category annotation information.

2. The training method for the prompt-free image semantic segmentation model according to claim 1, characterized in that, The image semantic segmentation model includes a classification module, which is used to generate category prompt information for the corresponding image samples based on the image fusion features. The classification module includes a multi-layer network constructed based on a trainable activation function matrix, and each layer of the network includes residual connections and convolutional alignment units.

3. The training method for the prompt-free image semantic segmentation model according to claim 2, characterized in that, The input of each layer of the network in the classification module is directly superimposed to the feature output of the corresponding layer through the residual connection, and after being superimposed with the feature output of the corresponding layer, it is processed by the convolutional alignment unit to become the final output of the corresponding layer.

4. The training method for the prompt-free image semantic segmentation model according to claim 1, characterized in that, The method includes: The general visual features and the general-specific features are fused using a feature stitching method; and / or, Based on the image fusion features, category hint information is generated for the corresponding image samples, specifically including: The image fusion features are reduced in dimensionality and integrated to obtain low-dimensional image fusion features; Based on the low-dimensional image fusion features, category hint information is generated for the corresponding image samples.

5. The training method for the prompt-free image semantic segmentation model according to claim 1, characterized in that, The category labeling information includes a true category mask, and the category hint information includes a predicted category mask; The preset loss function includes at least one of a first loss function sub-item, a second loss function sub-item, and a third loss function sub-item. The first loss function sub-item represents the degree of overlap between the true class mask and the predicted class mask. The second loss function sub-item represents the prediction error of the target image sample. The third loss function sub-item is used to represent the error between the degree of overlap between the true mask and the degree of overlap between the predicted mask. The target image samples include image samples whose classification confidence is lower than a preset threshold output by the image semantic segmentation model, and the degree of overlap of the real mask is calculated based on the degree of overlap between the real class mask and the predicted class mask.

6. A prompt-free image semantic segmentation method, characterized in that, The method includes: Obtain the image to be semantically segmented; The image to be semantically segmented is processed based on a prompt-free image semantic segmentation model to obtain a semantic segmentation mask; The image semantic segmentation model is trained by the training method described in any one of claims 1-5.

7. A training device for a prompt-free image semantic segmentation model, characterized in that, include: An acquisition module is used to acquire a training dataset, wherein the training dataset includes multiple image samples, and the image samples include category labeling information; The extraction and fusion module is used to extract the general visual features and general scene-specific features of the multiple image samples respectively and fuse them to obtain image fusion features; The prompting module is used to generate category prompting information for the corresponding image samples based on the image fusion features; The training module is used to adjust the parameters of the image semantic segmentation model based on a preset loss function until the training objective is achieved. The preset loss function includes an error loss term for the category prompt information and the category annotation information.

8. A prompt-free image semantic segmentation device, characterized in that, include: The acquisition module is used to acquire the image to be semantically segmented; The processing module is used to process the image to be semantically segmented based on the prompt-free image semantic segmentation model to obtain a semantic segmentation mask; The image semantic segmentation model is trained by the training method described in any one of claims 1-5.

9. An electronic device, characterized in that, include: At least one processor; as well as The memory stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the training method for the prompt-free image semantic segmentation model as described in any one of claims 1 to 5 or the prompt-free image semantic segmentation method as described in claim 6.

10. A machine-readable storage medium, characterized in that, It stores executable instructions that, when executed, cause the machine to perform the training method for the prompt-free image semantic segmentation model as described in any one of claims 1 to 5 or the prompt-free image semantic segmentation method as described in claim 6.

Images