Semantic segmentation method, apparatus, device, and storage medium

Abstract

The application relates to the technical field of computer vision, and discloses a semantic segmentation method and device, equipment and a storage medium, which comprise the following steps: preprocessing a to-be-segmented image to obtain a first input image and a second input image; inputting the first input image into a visual language model to obtain category prior information of the first input image; performing pixel feature extraction on the second input image to obtain a pixel feature map; determining classifier parameters through a dynamic analysis network based on the category prior information and the pixel feature map, and constructing a pixel-level classifier according to the classifier parameters; and performing semantic category discrimination on the pixel feature map through the pixel-level classifier to obtain a semantic segmentation result. The visual language model is used to obtain category prior information at the existence level from the input image, the category prior information is added to the construction process of the pixel-level classifier, the parameters of the classifier can adaptively change along with the semantic category of the input image, and the discrimination capability is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a semantic segmentation method, apparatus, device, and storage medium. Background Technology

[0002] In existing semantic segmentation methods, the classifier used to perform pixel-level discrimination is usually a fixed-parameter model that is learned all at once during the training phase. The discrimination process assumes that any input image may contain all semantic categories, making it difficult to adaptively optimize for the semantic composition of a specific image. As a result, in the pixel-level classification process, the absence of a category in the current image still interferes with the discrimination process, resulting in poor discrimination ability. Summary of the Invention

[0003] The main objective of this application is to provide a semantic segmentation method, apparatus, device, and storage medium, which aims to solve the technical problem of poor discriminative ability of pixel-level classifiers with fixed parameters in existing semantic segmentation methods.

[0004] To achieve the above objectives, this application proposes a semantic segmentation method, which includes:

[0005] The image to be segmented is preprocessed to obtain the first input image and the second input image; The first input image is input into the visual language model to obtain the category prior information of the first input image; Pixel feature extraction is performed on the second input image to obtain a pixel feature map; Based on the category prior information and the pixel feature map, the classifier parameters are determined by a dynamic parsing network, and a pixel-level classifier is constructed according to the classifier parameters. The pixel-level classifier is used to determine the semantic category of the pixel feature map to obtain the semantic segmentation result.

[0006] Optionally, the step of inputting the first input image into the visual language model to obtain the category prior information of the first input image includes: The first input image is input into the visual language model, and the existence score of each preset semantic category in the first input image is output. Construct a category prior vector based on the existence score; The class prior vectors are converted into class prior information in the form of continuous vectors based on the learnable matrix.

[0007] Optionally, constructing the category prior vector based on the existence score includes: Obtain the first semantic category from the preset semantic categories, where the existence score of the first semantic category is greater than a preset threshold; A preset number of second semantic categories are selected based on the existence score; Construct a category prior vector based on the first semantic category and / or the second semantic category.

[0008] Optionally, determining the classifier parameters through a dynamic parsing network based on the category prior information and the pixel feature map includes: Extract the global image representation of the second input image from the pixel feature map; The category prior information and the image global representation are concatenated to obtain the conditional vector; The conditional vector is input into the dynamic parsing network, which outputs the classifier parameters of the pixel-level classifier.

[0009] Optionally, the classifier parameters include weight parameters, bias parameters, and normalization parameters; The step of performing semantic category discrimination on the pixel feature map using the pixel-level classifier to obtain semantic segmentation results includes: Convert the pixel feature map into a pixel feature matrix; Based on the weight parameters and bias parameters of the pixel-level classifier, a linear transformation is performed on the pixel feature matrix to obtain a pixel-by-pixel classification score matrix. The classification score matrix is ​​normalized using the normalization parameter to obtain a pixel-by-pixel first category probability map; The first category probability map is upsampled to generate a second category probability map corresponding to the resolution of the image to be segmented; The semantic category of each pixel is determined based on the second category probability map, and the semantic segmentation result is obtained.

[0010] Optionally, the pixel-level classifier is a low-rank dynamic classifier; The step of performing a linear transformation on the pixel feature matrix based on the weight parameters and bias parameters of the pixel-level classifier to obtain a pixel-by-pixel classification score matrix includes: Obtain the preset static basis matrix; A weight matrix is ​​constructed using the static basis matrix and the weight parameters; Based on the weight matrix and the bias parameter, a linear transformation is performed on the pixel feature matrix to obtain a pixel-by-pixel classification score matrix.

[0011] Optionally, before preprocessing the image to be segmented to obtain the first input image and the second input image, the method further includes: Obtain training images and corresponding pixel-level ground truth values, and determine the true class prior information of the training images based on the pixel-level ground truth values; The training image is input into the visual language model to obtain the prior information of the predicted category of the training image; Based on the true category prior information and the predicted category prior information, the target category prior information is determined; Based on the prior information of the target category and the pixel feature map of the training image, a training pixel-level classifier is constructed, and the pixel feature map of the training image is classified by the training pixel-level classifier to obtain the predicted semantic segmentation result. Based on the predicted semantic segmentation results and the pixel-level labeled ground truth, the parameters of the visual language model and the dynamic parsing network are adjusted.

[0012] Furthermore, to achieve the above objectives, this application also proposes a semantic segmentation apparatus, which includes: The image processing module preprocesses the image to be segmented to obtain a first input image and a second input image; The category acquisition module is used to input the first input image into the visual language model and obtain the category prior information of the first input image; The feature extraction module is used to extract pixel features from the second input image to obtain a pixel feature map; The classifier construction module is used to determine classifier parameters through a dynamic parsing network based on the category prior information and the pixel feature map, and to construct a pixel-level classifier according to the classifier parameters; The category discrimination module is used to perform semantic category discrimination on the pixel feature map through the pixel-level classifier to obtain semantic segmentation results.

[0013] In addition, to achieve the above objectives, this application also proposes a semantic segmentation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the semantic segmentation method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the semantic segmentation method described above.

[0015] This application leverages the powerful image-level semantic understanding capabilities of Vision-Language Models (VLMs). It first obtains existence-level category priors from the input image and then explicitly injects these priors into the construction of the pixel-level classifier in a structured form. This transforms the classification decision of the pixel-level segmentation network from static, all-category discrimination to constrained, adaptive, dynamic discrimination without introducing conflicts between understanding and localization. Specifically, this application decomposes the pixel-level discrimination process in semantic segmentation into the synergy of two complementary capabilities: a pre-trained visual language model handles high-level semantic parsing, outputting the set of semantic categories contained in the image; a lightweight pixel encoder handles pixel feature extraction; and a small dynamic parsing network generates dynamic convolution parameters based on category priors and pixel features to construct a dynamic classifier oriented towards the semantic composition of the current image. Finally, pixel-by-pixel recognition is performed on the pixel features to output the segmentation result. The output of the visual language model is transformed from textual description into explicit priors that can be used to constrain the discrimination space. This prior knowledge is then integrated into the classifier parameters using a dynamic parameter generation mechanism, enabling the classifier to adapt to the semantic composition of the image during the inference stage and improving its discrimination ability. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the semantic segmentation method of this application; Figure 2 This is a schematic diagram illustrating the specific process of the semantic segmentation method in this application; Figure 3 This is a flowchart illustrating the second embodiment of the semantic segmentation method of this application; Figure 4 This is a flowchart illustrating the third embodiment of the semantic segmentation method of this application; Figure 5 This is a schematic diagram of the module structure of the semantic segmentation device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the semantic segmentation method in this application embodiment.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Existing semantic segmentation methods generally employ an end-to-end processing framework based on deep neural networks. The basic approach involves encoding pixel-level features of the input image, then using a classifier to determine the semantic category of each pixel's features, resulting in pixel-level semantic segmentation. In these methods, the classifier used for pixel-level discrimination is typically a fixed-parameter model learned all at once during the training phase and remains unchanged across all input images during the inference phase, representing a typical static classifier structure. The output dimension of this static classifier corresponds one-to-one with a pre-defined set of semantic categories, and its discrimination process assumes that any input image may contain all semantic categories.

[0023] In practical applications, an input image typically contains only a few categories from a predefined set of semantic categories, and the category combinations differ significantly between different images. Static classifiers mostly employ a fixed parameter structure learned all at once during the training phase. During the inference phase, they consistently use the complete set of categories as the discrimination space for classification decisions, failing to fully consider the significant differences in the semantic composition of input images. They cannot shrink or adjust the classification decision space based on the actual semantic categories present in the current input image. The decision boundary of a static classifier is uniformly determined across the entire data distribution and the entire category space, making it difficult to adaptively optimize for the semantic composition of a specific image. Consequently, during pixel-level classification, categories not present in the current image still interfere with the discrimination process, diluting the classifier's ability to distinguish actually present categories. Feature responses of the same semantic category in the current image are unlikely to form a consistent distribution, and the discrimination boundaries between different semantic categories are difficult to effectively strengthen within the image context.

[0024] In recent years, visual language models have rapidly developed in general visual understanding. Early models focused on text-image alignment and image-level classification, evolving to more sophisticated visual recognition, localization, and cross-modal reasoning, gradually showing a trend towards becoming a general-purpose visual intelligence foundation. On one hand, technologies represented by multimodal large models have achieved unified modeling and transfer across multiple scenarios, including single-image, multi-image, and video tasks, significantly reducing the development costs of dedicated models for different visual tasks. On the other hand, next-generation visual language models have continuously improved their capabilities in visual recognition, object localization, document parsing, and long-video understanding, giving them stronger generalization and stability in image-level semantic parsing tasks such as "determining what's in an image." Although visual language models have strong image-level understanding capabilities, there is a gap between understanding and localization when directly applied to pixel-level segmentation tasks; enhancing localization and segmentation capabilities would sacrifice their original semantic understanding advantages.

[0025] Therefore, this application provides a semantic segmentation method that combines large and small models. Instead of relying on a large visual language model to directly output pixel-level masks or undertake dense prediction tasks, it utilizes a pre-trained visual language model to perform overall semantic parsing of the input image, obtaining a set of semantic categories contained within the image. This set of semantic categories is then mapped to one-hot or multi-hot category prior vectors, which are used as conditional inputs in the subsequent pixel-level semantic segmentation process. This introduces computationally computable and structured image-level semantic prior information into the segmentation system. Furthermore, a dynamic parsing network is used to jointly model the category prior vectors with the pixel features output by the encoder, dynamically generating classifier parameters for pixel-level semantic classification. The dynamic classifier is then used to classify pixel features pixel by pixel, outputting semantic segmentation results. This allows the parameters of the pixel-level classifier to adaptively change according to the semantic category composition of the input image.

[0026] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, feature extraction, and program execution functions, such as a computer, or an electronic device capable of performing the above functions. The following description uses a semantic segmentation device as an example to illustrate this embodiment and the subsequent embodiments.

[0027] Based on this, embodiments of this application provide a semantic segmentation method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the semantic segmentation method of this application.

[0028] In this embodiment, the semantic segmentation method includes: Step S10: Preprocess the image to be segmented to obtain the first input image and the second input image.

[0029] It should be noted that the image to be segmented refers to the original image that requires pixel-level semantic segmentation, such as road images in autonomous driving scenarios, CT images in medical imaging, or remote sensing images. Preprocessing involves performing necessary format conversions, resizing, and normalization operations on the original image to adapt it to the input requirements of different subsequent modules.

[0030] Understandably, the first input image is the image input to the visual language model, and the second input image is the image input to the pixel feature encoder. In practical applications, the two can be the same image, i.e., using the same preprocessing parameters; or they can be images of different resolutions. For example, to balance the robustness of the visual language model's semantic parsing with the pixel encoder's sensitivity to detail, the first input image can be scaled or cropped at a larger scale to maintain semantic integrity, while the second input image retains a higher resolution to facilitate fine extraction of pixel-level features.

[0031] It should be understood that for the input image Preprocessing operations, including but not limited to normalization, scaling, cropping, padding, or combinations thereof, are not limited in this embodiment. The final input is then obtained for use by the visual language model and pixel encoder. and Both can be the same or different resolutions to balance semantic parsing robustness and pixel encoding accuracy.

[0032] Step S20: Input the first input image into the visual language model to obtain the category prior information of the first input image.

[0033] It should be noted that visual language models are pre-trained, large-scale deep learning models capable of jointly understanding images and text, such as CLIP, BERT, or their variants. Trained on massive amounts of image-text pairs, visual language models possess powerful image-level semantic understanding capabilities, enabling them to determine the objects or scenes present in an image. This embodiment can leverage this capability to extract semantic category information from the first input image, serving as prior guidance for subsequent segmentation.

[0034] Understandably, category prior information is used to indicate the actual semantic category in an image, and it serves as a constraint on the entire category space.

[0035] Specifically, will Input pre-trained visual language model This process obtains the set of semantic categories contained in the image. In this embodiment, this process can be abstracted as a mapping from the image to a category existence vector. Specifically, it can be implemented in any of the following ways: First, the visual language model directly outputs the set of categories. Then, the category prior embedding subsystem performs the mapping; secondly, the visual language model outputs the existence confidence vector for each category. Then through threshold or Select the set Third, the category name or description is used as a textual cue input to the visual language model to obtain an image-text matching score and form a... .

[0036] In the specific implementation process, a learnable matrix can be used to map the binary category prior vector into a continuous embedding vector, providing a structured conditional input for the subsequent dynamic parsing network, thus realizing the fusion of semantic prior and neural network computation. At this point, step S20 may include: The first input image is input into the visual language model, and the existence score of each preset semantic category in the first input image is output; a category prior vector is constructed based on the existence score; and the category prior vector is converted into category prior information in the form of a continuous vector based on the learnable matrix.

[0037] It should be noted that after the first input image is input into the visual language model, the model will output an existence score vector. ,in This represents the confidence level of the k-th preset semantic category appearing in the image. This score can be obtained in several ways.

[0038] As an optional implementation, the visual language model can employ a dual-encoder structure, comprising an image encoder and a text encoder. For each preset semantic category, its category name (e.g., "person") or a more detailed text description (e.g., "a person walking on the street") is input into the text encoder to obtain text features. Simultaneously, the first input image is input into the image encoder to obtain image features. Then, the cosine similarity between the image features and each text feature is calculated as the existence score for that category. The higher the similarity, the greater the likelihood that the image contains that category.

[0039] In another alternative implementation, the visual language model can add a multi-label classification head after the image encoder, directly outputting a probability value between 0 and 1 for each category as an existence score. This classification head can be fine-tuned on a specific dataset.

[0040] It should be noted that the category prior vector is a binary vector used to explicitly indicate whether each preset semantic category exists in the current image, and it forms the basis for subsequent parameterized representations. Its dimension is the same as the total number of preset semantic categories K. Each element in the vector corresponds to a category, with a value of 1 indicating that the category exists in the image and a value of 0 indicating that it does not exist.

[0041] Understandably, the core of constructing category prior vectors lies in transforming continuous existence scores into discrete binary indicators. In this embodiment, this transformation can be achieved through thresholding or TopM selection.

[0042] It should be noted that the category prior vector is a discrete binary vector with elements of 0 or 1. This discrete representation cannot be directly optimized by gradient descent, nor is it convenient to use as input to subsequent neural networks. Therefore, in this embodiment, a learnable embedding matrix can be introduced to map the discrete vector to a continuous vector, enabling it to participate in end-to-end training and inference.

[0043] In practical implementation, threshold-based filtering and TopM selection strategies can make the acquisition of category prior information more flexible and robust, adapting to different application scenarios and varying requirements for recall / precision. At this point, step S20 may include: Obtain a first semantic category from the preset semantic categories, wherein the existence score of the first semantic category is greater than a preset threshold; filter a preset number of second semantic categories based on the existence score; construct a category prior vector based on the first semantic category and / or the second semantic category.

[0044] It should be noted that the first semantic category refers to the set of categories with high confidence in the image, selected through a thresholding method. Preset threshold. It is a pre-set value used to determine whether the existence score of a category is high enough to confirm that the category exists in the image.

[0045] Specifically, for each preset semantic category Compare their existence scores With preset threshold The size. If If the threshold is reached, then the category is identified as the first semantic category, i.e., a candidate category existing in the image. The value of is typically between 0 and 1 (when the existence score has been normalized to probability), or it can be any real number (when the existence score is unnormalized logits). The choice of threshold can be adjusted according to the specific application scenario: in scenarios requiring high recall (such as in security monitoring where no suspicious targets should be missed), a lower threshold can be set to include more categories; in scenarios requiring high precision (such as in autonomous driving to avoid incorrect decisions due to misidentification), a higher threshold can be set to ensure that only categories with high confidence are considered.

[0046] It should be noted that the second semantic category refers to the set of categories with the highest existence scores selected by the TopM selection method. The preset number M is a pre-defined integer representing how many categories with the highest scores are desired to be selected as existence candidates from all categories.

[0047] Specifically, the existence scores of all preset semantic categories are sorted in descending order to obtain a ranking. The top M categories are selected as the second semantic category, i.e., the set of categories with the TopM scores. M can be a fixed value, such as set according to the average number of categories contained in each image based on the dataset statistics, such as M=5, which means that a maximum of 5 existing categories are considered for each image; M can also be dynamically changed, for example, adaptively determined according to the steepness of the score distribution, taking a smaller M when the score distribution is relatively concentrated, and taking a larger M when the score distribution is relatively dispersed. When the actual number of categories existing in the image varies greatly, the TopM method can ensure that M categories are always selected, avoiding the problem of selecting too many or too few categories that may occur with the threshold method.

[0048] It is understood that, in this embodiment, the first semantic category (threshold-based selection result) and the second semantic category (TopM-based selection result) can be used individually or in combination to jointly determine the final category prior vector. Which positions are set to 1 and which positions are set to 0?

[0049] As an alternative combined strategy, a threshold-based screening followed by Top-M truncation can be adopted: First, a first semantic category set is obtained based on the threshold method. If the cardinality (i.e., the number of categories) of the first semantic category set is greater than M, then the M categories with the highest scores are selected from the first semantic category set as positions for taking 1 in the final category prior vector; if it is less than or equal to M, then the entire first semantic category set is used as positions for taking 1. This method ensures that the selected categories have sufficient absolute confidence and limits the total number of selected categories, avoiding too many low-scoring categories being mistakenly selected.

[0050] As an alternative combined strategy, a TopM filtering followed by threshold elimination can be employed: First, a second semantic category set is obtained using the TopM method. Then, for each category in the second semantic category set, its score is checked to see if it is greater than or equal to a threshold. Only categories that satisfy the second semantic category set are retained as the final selection position. This method ensures that the selected categories are both relatively high-ranking and have sufficient absolute confidence, eliminating those that, although ranked high, still have very low absolute scores.

[0051] Specifically, in a preferred embodiment, the category existence score output by the visual language model is defined as...

[0052] in Indicate category The presence score in the image is used to construct the category prior vector. ,as follows:

[0053] in For the threshold, This is an indicator function. Alternatively, it can also be used... Choose a strategy:

[0054] in Return the highest score A collection of category indexes. Threshold. and It can be configured according to the application scenario. To enable category priors to serve as conditional inputs for subsequent networks, this application further provides a category prior embedding subsystem, which... Class prior information parameterized as a continuous vector Learnable matrices are used. :

[0055] This represents the category prior embedding dimension. It can be implemented directly using identity mapping or using a multilayer perceptron.

[0056] Step S30: Extract pixel features from the second input image to obtain a pixel feature map.

[0057] Specifically, the input image Input pixel feature encoder Obtain pixel feature maps

[0058] This represents the spatial resolution of the feature map. The encoder can be a convolutional network, a visual self-attention model, a state-space model, or a combination thereof, and can also include multi-scale structures such as a Feature Pyramid Network. If the encoder output is a multi-scale feature set... Then use the feature fusion module. Obtain uniform scale features This fusion method does not constitute a limitation on this application.

[0059] Step S40: Based on the category prior information and the pixel feature map, classifier parameters are determined by a dynamic parsing network, and a pixel-level classifier is constructed according to the classifier parameters.

[0060] It should be noted that a dynamic parsing network refers to a lightweight parameter generation network whose input is a conditional vector and whose output is a set of parameters for a pixel-level classifier. The structure of a dynamic parsing network can take the form of a multilayer perceptron, convolutional neural network, visual self-attention model, state-space model, hypernetwork, or a combination thereof, satisfying the requirement of outputting classifier parameters. And it can be used for pixel-level classification. For example... ,in For dynamically generated parts, when there are many classification header parameters, the generation difficulty can be reduced and the stability can be improved.

[0061] Understandably, classifier parameters refer to a set of learnable or generateable variables used to map pixel features to semantic category scores. In this invention, classifier parameters include at least weight parameters, and may also include bias parameters, normalization parameters, etc. These parameters are dynamically generated by the dynamic parsing network based on the conditional information of the current input image, rather than remaining fixed after training as in traditional methods. The specific form and dimensions of the classifier parameters depend on the classifier structure employed.

[0062] In the specific implementation process, prior category information can be fused with the global visual representation of the image and used as input to the dynamic parsing network. This allows the generated classifier parameters to better adapt to the visual features of the current image, improving the discrimination accuracy of the dynamic classifier. At this point, step S40 may include: Extract the global image representation of the second input image from the pixel feature map; concatenate the category prior information and the global image representation to obtain a condition vector; input the condition vector into the dynamic parsing network to output the classifier parameters of the pixel-level classifier.

[0063] It's important to note that global image representation is a holistic description of the entire image content, used to supplement potentially missing visual context in category prior information. While category prior information already indicates which semantic categories exist in the image, the visual representation of the same category can differ across different images. For example, "vehicle" can refer to a sedan, truck, or bicycle, and "road" can be a highway, a country lane, or a city street. Global image representation aims to capture these visual differences, enabling the generated classifier parameters to better adapt to the specific visual features of the current image.

[0064] Specifically, in order to synchronously consider the pixel representation of the image during dynamic parameter generation, this embodiment allows the dynamic parsing network to receive prior category information embedded prior to the prior. In addition, it also receives pixel feature maps Extracted global image representation Condition vector It contains information on two levels: Part of the instruction tells the dynamic parsing network "what categories should be in the image". Part of the information tells the dynamic parsing network "what the image looks like overall." The combination of these two aspects allows the subsequently generated classifier parameters to be aware of both semantic-level category constraints and visual-level appearance changes, resulting in more accurate classification decision boundaries. In a preferred embodiment, Global Average Pooling (GAP) is used to obtain:

[0065] Then and Concatenate to form a conditional vector:

[0066] Inapplicable In an alternative implementation, the condition vector can be made Dynamically parsed networks by The input is the set of parameters for the pixel-level classifier. Formal representation:

[0067] in, The output classifier parameters must include at least the classifier's weight parameters, which may include, but are not limited to, the following: weight matrix. (Used to map pixel feature vectors to the category score space), bias vector (Add a learnable bias term for each category) and normalization layer parameters (including scaling and translation factors if the classifier includes batch normalization or layer normalization).

[0068] As an optional implementation method, dynamic resolution network A multilayer perceptron can be used, consisting of multiple fully connected layers stacked together, each followed by a non-linear activation function. The input layer receives a conditional vector. The intermediate layers progressively transform the features, and the dimension of the output layer equals the total number of parameters required for generation. For example, if the weight matrix of a 1×1 convolutional classifier needs to be generated... and bias vector Then the output layer dimension should be D×K+K. The original vector output by the network is then reshaped to obtain the weight matrix and bias vector.

[0069] Step S50: The pixel feature map is semantically classified using the pixel-level classifier to obtain the semantic segmentation result.

[0070] It is understandable that classification is performed based on the feature vector of each spatial location in the pixel feature map of the dynamic pixel-level classifier to determine the semantic category to which each pixel belongs, and finally a pixel-level semantic label map aligned with the input image space can be output.

[0071] In one example, reference Figure 2 , Figure 2 This is a schematic diagram illustrating the specific workflow of the semantic segmentation method in this application. After processing, the input image is fed into both the encoder and the visual language model. The encoder extracts pixel features, and the visual language model performs image classification using prompts, generating multi-label image classification results such as roads, sidewalks, buildings, and fences. The multi-label image classification results are converted into one-hot encodings and concatenated with the pixel features obtained through global mean pooling along the channel dimension. The concatenated result is input into a dynamic parsing network, which outputs classifier parameters to construct a classifier. Based on the classifier, semantic segmentation is performed on the pixel features of the identity mapping, and the prediction result is output.

[0072] Specifically, one example of the prompt word in this embodiment is: "This image is from [...]" The scenario defines the following semantic categories: , , Now, please identify all semantic categories appearing in the image from the list of categories above, treating this task as a multi-label image classification task. You must carefully examine the image and determine whether each category exists one by one. Consider all [the categories independently before giving the final answer]. [Number of categories]. Only categories that are clearly and unambiguously visible in the image should be included; do not infer any categories that are not visually observable. The final output should be a list of category names only.

[0073] In this embodiment, the existence-level category prior is obtained from the input image through a visual language model, and then the prior is explicitly injected into the construction process of the pixel-level classifier in a structured form. This allows the classification decision of the pixel-level segmentation network to be transformed from static discrimination of all categories to constrained and adaptive dynamic discrimination without introducing conflicts between understanding and localization, thereby improving the inference ability of the classifier.

[0074] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the semantic segmentation method of this application. Based on the first embodiment described above, a second embodiment of the semantic segmentation method of this application is proposed.

[0075] In the second embodiment, the classifier parameters include weight parameters, bias parameters, and normalization parameters. Step S50 includes: Step S501: Convert the pixel feature map into a pixel feature matrix.

[0076] It should be noted that the pixel feature map is a three-dimensional feature tensor output by the pixel feature encoder. ,in and This represents the spatial height and width of the feature map. This represents the number of feature channels (i.e., the dimension of the feature vector at each spatial location). This feature map preserves the spatial topology of the input image, with each spatial location... Corresponding to a D-dimensional feature vector It encodes the local visual information of the region.

[0077] To facilitate subsequent matrix operations, the three-dimensional feature map needs to be converted into a two-dimensional matrix form. This conversion essentially involves expanding the spatial dimension, treating each spatial location as an independent sample.

[0078] Specifically, pixel feature maps Flattening the matrix in the spatial dimension yields a two-dimensional matrix. ,in This represents the total number of pixel locations in the feature map. The number of rows N in the matrix equals the total number of spatial locations in the feature map, with each row corresponding to a specific spatial location. The number of columns D in the matrix equals the number of feature channels, with each column corresponding to a feature dimension. The elements in the matrix represent the response value of the nth spatial location on the dth feature channel.

[0079] Step S502: Based on the weight parameters and bias parameters of the pixel-level classifier, perform a linear transformation on the pixel feature matrix to obtain a pixel-by-pixel classification score matrix.

[0080] Understandably, for dynamics Convolutional classifiers flatten pixel features according to their positions. ,in Dynamically analyze the network output weight matrix and bias ,Right now The logit for each pixel position is calculated as follows:

[0081] The final result This is the classification score matrix, and its dimensions are... ,in This represents the total number of pixel locations. This represents the raw score (logit) of the nth pixel belonging to the kth category.

[0082] In practical implementation, the dynamic weight matrix can be decomposed into the product of a static basis matrix and a dynamic coefficient matrix, reducing the number of parameters that the dynamic analytical network needs to generate, lowering the risk of overfitting, and improving the training stability and inference efficiency of the model. At this point, step S502 may include: Obtain a preset static basis matrix; construct a weight matrix using the static basis matrix and the weight parameters; perform a linear transformation on the pixel feature matrix based on the weight matrix and the bias parameters to obtain a pixel-by-pixel classification score matrix.

[0083] It is understandable that, in another alternative implementation, if a low-rank dynamic classifier structure is used, the weight matrix is ​​derived from the static basis matrix. and dynamic coefficient matrix The product of , i.e. At this point, the calculation of the linear transformation can be optimized using the associative law of matrix multiplication to avoid explicitly constructing the complete W matrix, thereby reducing computational load and memory usage.

[0084] in, This allows the production quantity of dynamic parameters to be reduced from... Reduce to The advantage of this decomposition method lies in the fact that the static basis matrix A learns a general feature projection subspace, while the dynamic coefficient matrix B adaptively adjusts the representation of each class in this subspace based on the prior information of the current image's categories. This ensures the classifier's adaptability to specific images while significantly reducing the number of dynamically generated parameters.

[0085] Step S503: Normalize the classification score matrix using the normalization parameter to obtain a pixel-by-pixel first category probability map.

[0086] Step S504: Upsample the first category probability map to generate a second category probability map corresponding to the resolution of the image to be segmented.

[0087] Step S505: Determine the semantic category of each pixel based on the second category probability map to obtain the semantic segmentation result.

[0088] It should be noted that, as an infrastructure approach, if the normalization parameter is empty, then only the standard Softmax function is used for normalization. The Softmax function transforms the score vector into a probability distribution, and its mathematical expression is:

[0089] Then Restore to tensor space If you need to output a result with the same resolution as the original image, you can use an upsampling operator. get The final semantic label for the pixel is:

[0090] in This represents the predicted probability that the nth pixel belongs to the kth class. The argmax operation returns the index of the class that maximizes the probability. Combining the labels at all positions (i,j) yields the final semantic segmentation result image. The resulting image has the same dimensions as the original input image, and the value at each pixel location is the semantic category number to which that pixel belongs.

[0091] In another implementation, if the generated classifier parameters include normalization layer parameters (such as batch normalization or layer normalization parameters), these normalization layers need to be applied after the linear transformation and before the Softmax normalization to standardize the score matrix. This embodiment does not impose any restrictions on this.

[0092] In this embodiment, dynamically generated classifier parameters are applied to the standard classification calculation process, and upsampling is used to achieve spatial alignment from feature map resolution to original image resolution, so that the output semantic segmentation result can be mathematically strictly aligned with the input image.

[0093] Reference Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the semantic segmentation method of this application. Based on the above embodiments, a third embodiment of the semantic segmentation method of this application is proposed.

[0094] In the third embodiment, before step S10, the method further includes: Step S01: Obtain the training image and the corresponding pixel-level labeled ground truth value, and determine the true category prior information of the training image based on the pixel-level labeled ground truth value.

[0095] It should be noted that training images are image samples selected from the training dataset for optimizing model parameters. The training dataset typically contains a large number of diverse images, covering various scenes and semantic categories, to ensure that the trained model has good generalization ability. Training images can be original images or images that have undergone data augmentation (such as random cropping, rotation, color jittering, etc.).

[0096] As can be understood, pixel-level ground truth refers to a label map with the exact same spatial resolution as the training image. Each pixel in the label map is assigned a specific category label, representing the true semantic category to which that pixel belongs. Typically generated by manual annotation, it serves as a standard supervisory signal in semantic segmentation tasks. Ground truth class prior information is indicative information derived from pixel-level ground truth annotations, reflecting which semantic categories the training image actually contains. This is achieved by traversing all pixels in the pixel-level ground truth annotations, recording all occurrences of category labels, and then constructing a binary vector corresponding to the entire set of preset semantic categories.

[0097] In an optional implementation, if the preset semantic category includes the background class, and there may be some unlabeled areas in the training image (such as blurred boundaries or areas ignored by the annotator), these areas can be excluded when calculating the prior information of the true category, and only the categories within the valid labeled areas can be considered.

[0098] Step S02: Input the training image into the visual language model to obtain the prior information of the predicted category of the training image.

[0099] As an optional implementation, the visual language model employs a dual-encoder structure, comprising an image encoder and a text encoder. For each preset semantic category, its category name or a more detailed textual description is input into the text encoder to obtain the corresponding text feature vector. Simultaneously, the training image is input into the image encoder to obtain an image feature vector. Then, the similarity (e.g., cosine similarity) between the image features and each text feature is calculated as the existence score for that category. After obtaining the existence score vector, it can be further converted into a binary predicted category prior vector. .

[0100] Step S03: Determine the target category prior information based on the true category prior information and the predicted category prior information.

[0101] It is understandable that when a visual language model is involved in fine-tuning, a true class prior is constructed from labeled data during the training phase. The predicted category prior information output by the visual language model Participate in training together. A category exists in the label if and only if One implementation method is to use during training. As a conditional input, to avoid early noisy prior information affecting model convergence, the visual language model's predictions are used during inference. Another implementation combines the two to obtain prior information about the target category:

[0102] In the early stages of training, set up Or close to 1, making the target prior completely or primarily dependent on the true prior, ensuring stability during training. This can be gradually reduced in the later stages of training. This ensures that the target prior is entirely determined by the prediction prior, which is completely consistent with the working conditions of the reasoning stage.

[0103] Step S04: Based on the prior information of the target category and the pixel feature map of the training image, construct a training pixel-level classifier, and classify the pixel feature map of the training image using the training pixel-level classifier to obtain the predicted semantic segmentation result.

[0104] It is understandable that, after determining the prior information of the target category... Then, the prior information of the categories is transformed into continuous vectors using a learnable matrix. This is used for building subsequent dynamic classifiers.

[0105] Step S05: Based on the predicted semantic segmentation result and the pixel-level labeled ground truth, adjust the parameters of the visual language model and the dynamic parsing network.

[0106] Specifically, supervised learning can be used to train the pixel feature encoder and the dynamic parsing network, while the visual language model can be fixed or fine-tuned. Indicates the input image The ground-truth label is obtained using the pixel-wise cross-entropy loss function:

[0107] If class imbalance exists, a weighted cross-entropy loss function or a Focal Loss-like loss function can also be used. The selection of the loss function does not constitute a limitation of this application.

[0108] The training objective of the dynamic parsing network is consistent with the segmentation loss. A regularization term can also be added to constrain the magnitude of the dynamic parameters, such as:

[0109] in, This represents the regularization coefficient. For low-rank dynamic classification heads, the dynamic parameters can be adjusted. Apply norm constraints to improve generalization ability.

[0110] During the inference or deployment phase, the process is consistent with the training phase, and the category prior is automatically generated by the visual language model from the input image. Specifically, the process involves: input image... Obtained through visual language model And generate Obtained through category prior embedding Simultaneously, the input image is obtained through a pixel feature encoder. and its global representation Dynamic parsing network based on Generate dynamic classifier parameters The dynamic classifier performs pixel-by-pixel classification of pixel features to obtain... The final segmentation result is obtained by upsampling and aligning the input image. In terms of deployment, the visual language model and the segmentation network can be executed on the same computing node, or they can be deployed separately in an edge-cloud collaborative scenario: the visual language model generates category priors on the cloud side, while the dynamic classifier construction and pixel segmentation are completed at the edge or on the device side; the category prior vectors... or its embedding Transmission bandwidth overhead and or It is proportional to the size of the image and is usually much smaller than the size of the entire image or high-dimensional feature map being transmitted.

[0111] In this embodiment, ideal priors are constructed using real annotations during the training phase to guide model learning, and these priors can be fused with the predictive priors of the visual language model. This allows for effective parameter optimization of the pixel encoding network, the dynamic parsing network, and even the visual language model itself while ensuring training stability.

[0112] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the semantic segmentation method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0113] This application also provides a semantic segmentation apparatus, please refer to... Figure 5 The semantic segmentation device includes: Image processing module 10 preprocesses the image to be segmented to obtain a first input image and a second input image; The category acquisition module 20 is used to input the first input image into the visual language model and obtain the category prior information of the first input image; Feature extraction module 30 is used to extract pixel features from the second input image to obtain a pixel feature map; The classifier construction module 40 is used to determine classifier parameters through a dynamic parsing network based on the category prior information and the pixel feature map, and to construct a pixel-level classifier according to the classifier parameters; The category discrimination module 50 is used to perform semantic category discrimination on the pixel feature map through the pixel-level classifier to obtain semantic segmentation results.

[0114] The semantic segmentation apparatus provided in this application, employing the semantic segmentation method described in the above embodiments, can solve the technical problem of poor discriminative ability of pixel-level classifiers with fixed parameters in existing semantic segmentation methods. Compared with the prior art, the beneficial effects of the semantic segmentation apparatus provided in this application are the same as those of the semantic segmentation method provided in the above embodiments, and other technical features in the semantic segmentation apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0115] This application provides a semantic segmentation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the semantic segmentation method in Embodiment 1 above.

[0116] The following is for reference. Figure 6The diagram illustrates a structural schematic of a semantic segmentation device suitable for implementing embodiments of this application. The semantic segmentation device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The semantic segmentation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0117] like Figure 6 As shown, the semantic segmentation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the semantic segmentation device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the semantic segmentation device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows semantic segmentation devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0118] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0119] The semantic segmentation device provided in this application, employing the semantic segmentation method described in the above embodiments, can solve the technical problem of poor discriminative ability of pixel-level classifiers with fixed parameters in existing semantic segmentation methods. Compared with the prior art, the beneficial effects of the semantic segmentation device provided in this application are the same as those of the semantic segmentation method provided in the above embodiments, and other technical features of this semantic segmentation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0120] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0121] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0122] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the semantic segmentation method in the above embodiments.

[0123] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0124] The aforementioned computer-readable storage medium may be included in the semantic segmentation device; or it may exist independently and not be assembled into the semantic segmentation device.

[0125] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the semantic segmentation device, cause the semantic segmentation device to perform the semantic segmentation method described above.

[0126] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0129] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described semantic segmentation method. This addresses the technical problem of poor discriminative ability of pixel-level classifiers with fixed parameters in existing semantic segmentation methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the semantic segmentation method provided in the above embodiments, and will not be elaborated upon here.

[0130] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A semantic segmentation method, characterized in that, The semantic segmentation method includes: The image to be segmented is preprocessed to obtain the first input image and the second input image; The first input image is input into the visual language model to obtain the category prior information of the first input image; Pixel feature extraction is performed on the second input image to obtain a pixel feature map; Based on the category prior information and the pixel feature map, the classifier parameters are determined by a dynamic parsing network, and a pixel-level classifier is constructed according to the classifier parameters. The pixel-level classifier is used to determine the semantic category of the pixel feature map to obtain the semantic segmentation result.

2. The semantic segmentation method as described in claim 1, characterized in that, The step of inputting the first input image into the visual language model and obtaining the category prior information of the first input image includes: The first input image is input into the visual language model, and the existence score of each preset semantic category in the first input image is output. Construct a category prior vector based on the existence score; The class prior vectors are converted into class prior information in the form of continuous vectors based on the learnable matrix.

3. The semantic segmentation method as described in claim 2, characterized in that, The step of constructing the category prior vector based on the existence score includes: Obtain the first semantic category from the preset semantic categories, where the existence score of the first semantic category is greater than a preset threshold; A preset number of second semantic categories are selected based on the existence score; Construct a category prior vector based on the first semantic category and / or the second semantic category.

4. The semantic segmentation method as described in claim 1, characterized in that, The step of determining classifier parameters through a dynamic parsing network based on the category prior information and the pixel feature map includes: Extract the global image representation of the second input image from the pixel feature map; The category prior information and the image global representation are concatenated to obtain the conditional vector; The conditional vector is input into the dynamic parsing network, which outputs the classifier parameters of the pixel-level classifier.

5. The semantic segmentation method according to any one of claims 1 to 4, characterized in that, The classifier parameters include weight parameters, bias parameters, and normalization parameters; The step of performing semantic category discrimination on the pixel feature map using the pixel-level classifier to obtain semantic segmentation results includes: Convert the pixel feature map into a pixel feature matrix; Based on the weight parameters and bias parameters of the pixel-level classifier, a linear transformation is performed on the pixel feature matrix to obtain a pixel-by-pixel classification score matrix. The classification score matrix is ​​normalized using the normalization parameter to obtain a pixel-by-pixel first category probability map; The first category probability map is upsampled to generate a second category probability map corresponding to the resolution of the image to be segmented; The semantic category of each pixel is determined based on the second category probability map, and the semantic segmentation result is obtained.

6. The semantic segmentation method as described in claim 5, characterized in that, The pixel-level classifier is a low-rank dynamic classifier; The step of performing a linear transformation on the pixel feature matrix based on the weight parameters and bias parameters of the pixel-level classifier to obtain a pixel-by-pixel classification score matrix includes: Obtain the preset static basis matrix; A weight matrix is ​​constructed using the static basis matrix and the weight parameters; Based on the weight matrix and the bias parameter, a linear transformation is performed on the pixel feature matrix to obtain a pixel-by-pixel classification score matrix.

7. The semantic segmentation method according to any one of claims 1 to 4, characterized in that, Before preprocessing the image to be segmented to obtain the first input image and the second input image, the process further includes: Obtain training images and corresponding pixel-level ground truth values, and determine the true class prior information of the training images based on the pixel-level ground truth values; The training image is input into the visual language model to obtain the prior information of the predicted category of the training image; Based on the true category prior information and the predicted category prior information, the target category prior information is determined; Based on the prior information of the target category and the pixel feature map of the training image, a training pixel-level classifier is constructed, and the pixel feature map of the training image is classified by the training pixel-level classifier to obtain the predicted semantic segmentation result. Based on the predicted semantic segmentation results and the pixel-level labeled ground truth, the parameters of the visual language model and the dynamic parsing network are adjusted.

8. A semantic segmentation device, characterized in that, The device includes: The image processing module preprocesses the image to be segmented to obtain a first input image and a second input image; The category acquisition module is used to input the first input image into the visual language model and obtain the category prior information of the first input image; The feature extraction module is used to extract pixel features from the second input image to obtain a pixel feature map; The classifier construction module is used to determine classifier parameters through a dynamic parsing network based on the category prior information and the pixel feature map, and to construct a pixel-level classifier according to the classifier parameters; The category discrimination module is used to perform semantic category discrimination on the pixel feature map through the pixel-level classifier to obtain semantic segmentation results.

9. A semantic segmentation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the semantic segmentation method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the semantic segmentation method as described in any one of claims 1 to 7.

Images