Image processing method and terminal device

Abstract

This application discloses an image processing method and a terminal device for achieving semantic recognition of images. The image processing method includes a semantic segmentation model comprising a visual encoder and a clustering head. The method includes: inputting the image to be predicted into the visual encoder to obtain pixel-level embedding features; inputting the pixel-level embedding features into the clustering head to obtain clustering results; pooling the pixel-level embedding features based on the clustering results to obtain visual embedding features corresponding to multiple image regions; and performing semantic segmentation on the multiple image regions based on the visual embedding features and text embedding features corresponding to each image region to obtain a semantic segmentation result for each image region. The text embedding features are obtained based on predefined text categories.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method and terminal device. Background Technology

[0002] Semantic segmentation is a task that takes a planar image as input and assigns a class label to each pixel. Current segmentation datasets require pixel-by-pixel annotation, which is costly to do manually and often contains only a limited number of semantic categories; for example, the PASCAL VOC dataset contains only 20 categories.

[0003] However, downstream complex application scenarios require segmentation models to complete open-set semantic segmentation tasks without specific category labeling. Open-set semantic segmentation technology has a wide range of applications, covering most unlabeled or poorly labeled semantic segmentation tasks. For example, semantic segmentation of images collected in autonomous driving application scenarios to identify non-whitelisted objects on the road (corner case); semantic segmentation of device images of fixed positions or regions of interest in China Compulsory Certification (3C) production lines; and semantic segmentation of images taken in terminal photography scenarios.

[0004] Current research on semantic segmentation mainly focuses on supervised semantic segmentation, which requires supervised training on finely labeled artificial datasets. The trained neural network models are usually only applicable to the categories or scenes covered by the dataset, and their generalization performance is poor, which is far from achieving the goal of correctly processing real-world images.

[0005] To bridge the gap between supervised semantic segmentation and real-world applications, several basic zero-shot semantic segmentation models have been proposed. However, most of these methods explore deep semantic relationships between unseen and seen classes, requiring large amounts of labeled data and a wider variety of seen class categories. The training datasets for these methods are often limited to a single dataset or datasets with specific content, making them prone to overfitting to the corresponding scene or category. They also fail to effectively segment diverse categories in the real world, resulting in unrecognized semantic results in images. Summary of the Invention

[0006] This application provides an image processing method and a terminal device for realizing semantic recognition of images.

[0007] To address the aforementioned technical problems, this application provides the following technical solutions:

[0008] Firstly, embodiments of this application provide an image processing method. The semantic segmentation model includes a visual encoder and a clustering head. The method includes: inputting an image to be predicted into the visual encoder to obtain pixel-level embedding features; inputting the pixel-level embedding features into the clustering head to obtain clustering results; pooling the pixel-level embedding features according to the clustering results to obtain visual embedding features corresponding to multiple image regions; performing semantic segmentation on the multiple image regions according to the visual embedding features and text embedding features corresponding to the multiple image regions to obtain a semantic segmentation result corresponding to each image region; the text embedding features are obtained based on predefined text categories. In the above scheme, embodiments of this application employ visual-based contrast, allowing the semantic segmentation model to better learn fine-grained visual embedding features. Furthermore, it only requires predefined text categories to encode text embedding features, eliminating the need for manually labeled data. Therefore, it significantly improves the semantic segmentation effect and quickly achieves semantic recognition of images.

[0009] In one possible implementation, inputting predefined text categories into the text encoder to obtain text embedding features includes: combining predefined text categories and predefined prompt templates to obtain text; and inputting the text into the text encoder to obtain text embedding features corresponding to multiple text categories. In the above scheme, prompt templates are applied in the field of natural language research and are used to form sentences resembling natural language together with category names, serving as a means of cross-modal pre-training. By combining prompt templates and text categories, text can be obtained quickly.

[0010] In one possible implementation, the step of semantically segmenting the multiple image regions based on their respective visual embedding features and text embedding features to obtain a semantic segmentation result for each image region includes: calculating the similarity between the visual embedding features of each image region and the text embedding features of the multiple text categories to obtain multiple similarity values ​​for each image region; and using the text category corresponding to the maximum similarity value among the multiple similarity values ​​as the semantic segmentation result for each image region. In the above scheme, multiple similarity values ​​for each image region can be obtained by calculating the similarity between the visual embedding features and text embedding features of different image regions, and then selecting the maximum similarity value to obtain the semantic segmentation result for each image region.

[0011] In one possible implementation, the method further includes: training the semantic segmentation model using image-text pairs without labeled text categories, and then outputting the trained semantic segmentation model. In the above scheme, image-text pairs without labeled text categories are first obtained; for example, these image-text pairs can be image-text pair data from the internet, without any fine-grained annotation. This embodiment can use these image-text pairs as training data to train the original semantic segmentation model, thereby completing the semantic segmentation training and outputting the trained semantic segmentation model.

[0012] In one possible implementation, training the semantic segmentation model using image-text pairs without labeled text categories includes: obtaining the original image and corresponding original text from the image-text pairs; performing image data augmentation on the original image to obtain multiple local images corresponding to the original image; inputting the original image and the multiple local images into the visual encoder to obtain global visual embedding features and local visual embedding features; performing pooling processing on the global visual embedding features and the local visual embedding features respectively to obtain pooled global visual embedding features and pooled local visual embedding features; embedding the original text... Cross-modal comparison supervision is performed on the features and the pooled global visual embedding features to obtain cross-modal comparison results. The original text embedding features are obtained by feature extraction from the original text through the semantic segmentation model. The global visual embedding features and the local visual embedding features are input into the clustering head, and the clustering head calculates the maximum mutual information of the clustering results distribution of the same pixel between the global visual embedding features and the local visual embedding features. By optimizing the maximum mutual information of the cross-modal comparison results and the clustering results distribution of the same pixel, the semantic segmentation model is trained, and training ends when the semantic segmentation model converges. In the above scheme, the maximum mutual information of the cross-modal comparison results and the clustering results distribution of the same pixel is generated and optimized respectively. By optimizing the maximum mutual information of the cross-modal comparison results and the clustering results distribution of the same pixel, the semantic segmentation model can be trained, and the convergence of the semantic segmentation model is judged at each optimization. Finally, the training of the semantic segmentation model can be terminated.

[0013] In one possible implementation, the step of training the semantic segmentation model by optimizing the mutual information of the cross-modal contrast results and the distribution of clustering results of the same pixel to maximize mutual information, and ending training when the semantic segmentation model converges, includes: performing visual contrast supervision on the pooled global visual embedding features and the pooled local visual embedding features to obtain visual contrast results; optimizing the mutual information of the visual contrast results, the cross-modal contrast results, and the distribution of clustering results of the same pixel to maximize mutual information to train the semantic segmentation model, and ending training when the semantic segmentation model converges. In the above scheme, visual contrast supervision can be introduced into the visual encoder, which can improve the training effect of the semantic segmentation model. During the model training phase, the semantic segmentation model can be trained by optimizing the mutual information of the visual contrast results, the cross-modal contrast results, and the distribution of clustering results of the same pixel to maximize mutual information.

[0014] In one possible implementation, the step of performing visual contrast supervision on the pooled global visual embedding features and the pooled local visual embedding features to obtain a visual contrast result includes: performing a fully connected mapping on the pooled global visual embedding features and the pooled local visual embedding features to obtain a fully connected mapping result; calculating the semantic consistency between the global visual embedding features and the local visual embedding features based on the fully connected mapping result, and calculating a loss function to obtain a visual contrast result including the loss function. In the above scheme, the visual encoder can generate a fully connected mapping result based on the pooled global visual embedding features and the pooled local visual embedding features, and calculate a loss function for visual contrast supervision based on the semantic consistency between the global visual embedding features and the local visual embedding features. For example, cross-entropy can be used as the loss function to obtain the visual contrast result.

[0015] In one possible implementation, the cross-modal comparison supervision of the original text embedding features and the pooled global visual embedding features to obtain a cross-modal comparison result includes: when there are multiple image-text pairs, calculating the cross-modal supervision loss function of the multiple image-text pairs based on the original text embedding features and the pooled global visual embedding features; and calculating the average value of the cross-modal supervision loss functions of the multiple image-text pairs to obtain a cross-modal comparison result. In the above scheme, the visual encoder can calculate the cross-modal supervision loss function of each image-text pair, and the average value of the cross-modal supervision loss functions of multiple image-text pairs can be used to obtain a cross-modal comparison result. In this embodiment, cross-modal supervision can align the embedding spaces of visual features and text features.

[0016] In one possible implementation, the clustering head includes an online clustering head. In this scheme, the online clustering head can dynamically divide pixel-level embedded features into different semantic regions, and during the inference phase, it uses region pooling for alignment, resulting in smoother semantic segmentation results.

[0017] In one possible implementation, the semantic segmentation results can be used in the fields of autonomous driving, product and device defect detection, or terminal image classification and recognition.

[0018] Specifically, the semantic segmentation results are used to identify non-whitelisted objects on the road surface in autonomous driving systems; or for defect detection of product components; or for classification, image editing, and text-based image retrieval and analysis of images captured by terminal devices. In the above scheme, semantic segmentation can be applied to the needs of open-set semantic segmentation, identifying images in the field of autonomous driving, detecting defects in product components, analyzing images captured by terminal devices, assisting in image editing, and conducting text-based image retrieval and analysis.

[0019] Secondly, embodiments of this application also provide a terminal device, wherein the semantic segmentation model includes: a visual encoder and a clustering head, wherein the terminal device includes:

[0020] A visual encoding module is used to input the image to be predicted into the visual encoder to obtain pixel-level embedding features;

[0021] The clustering module is used to input the pixel-level embedded features into the clustering head to obtain the clustering results;

[0022] The pooling module is used to pool the pixel-level embedding features based on the clustering results to obtain visual embedding features corresponding to multiple image regions respectively.

[0023] The semantic segmentation module is used to perform semantic segmentation on the multiple image regions based on the visual embedding features and text embedding features corresponding to the multiple image regions respectively, so as to obtain the semantic segmentation result corresponding to each image region; the text embedding features are obtained based on predefined text categories.

[0024] In the second aspect of this application, the constituent modules of the terminal device may also perform the steps described in the first aspect and various possible implementations, as detailed in the foregoing description of the first aspect and various possible implementations.

[0025] Thirdly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect above.

[0026] Fourthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect above.

[0027] Fifthly, embodiments of this application provide a communication device, which may include entities such as terminal devices or chips. The communication device includes: a processor and a memory; the memory is used to store instructions; the processor is used to execute the instructions in the memory, causing the communication device to perform the method as described in any one of the preceding first aspects.

[0028] Sixthly, this application provides a chip system including a processor for supporting a terminal device in implementing the functions involved in the above aspects, such as transmitting or processing data and / or information involved in the above methods. In one possible design, the chip system further includes a memory for storing necessary program instructions and data for the terminal device. This chip system may be composed of chips or may include chips and other discrete devices.

[0029] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0030] In this embodiment, the semantic segmentation model includes a visual encoder and a clustering head. The image to be predicted is input into the visual encoder to obtain pixel-level embedding features; the pixel-level embedding features are input into the clustering head to obtain clustering results; the pixel-level embedding features are pooled according to the clustering results to obtain visual embedding features corresponding to multiple image regions; semantic segmentation is performed on multiple image regions according to the visual embedding features and text embedding features corresponding to the multiple image regions to obtain semantic segmentation results corresponding to each image region; the text embedding features are obtained based on predefined text categories. This embodiment employs a visual-based contrast model, which can better learn fine-grained visual embedding features and only requires predefined text categories to encode text embedding features, eliminating the need for manually labeled data. Therefore, it significantly improves the semantic segmentation effect and quickly achieves semantic recognition of images. Attached Figure Description

[0031] Figure 1 A schematic diagram illustrating the implementation principle of an image processing method provided in an embodiment of this application;

[0032] Figure 2 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0033] Figure 3 A schematic diagram illustrating the training process of the semantic segmentation model provided in this application embodiment;

[0034] Figure 4 This is a schematic diagram of the inference process of the semantic segmentation model provided in the embodiments of this application;

[0035] Figure 5 This is a schematic diagram of the composition structure of a terminal device provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of the composition structure of another terminal device provided in an embodiment of this application. Detailed Implementation

[0037] This application provides an image processing method and a terminal device for realizing semantic recognition of images.

[0038] The embodiments of this application will now be described with reference to the accompanying drawings.

[0039] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0040] Semantic segmentation involves taking a planar image as input and assigning a class label to each pixel. Current segmentation datasets often contain only a limited number of semantic categories (e.g., the PASCAL VOC dataset contains only 20 categories) due to the high cost of manual annotation (pixel-by-pixel annotation is time-consuming and expensive). However, complex downstream applications require segmentation models to perform open-set segmentation tasks without specific class labels. Therefore, how to build models with open-set segmentation capabilities has become a major focus for both academia and industry. Open-set semantic segmentation techniques have wide applications, covering most unlabeled or poorly labeled semantic segmentation tasks. For example, in one scenario, the segmentation or collection of non-whitelisted objects (corner cases) on the road surface (e.g., garbage bags, stones, branches, etc.) in autonomous driving systems plays a crucial role in the safety of these systems. For example, in China Compulsory Certification (3C) production lines, the presence, absence, and errors of components (such as resistors, capacitors, inductors, connectors, screws, etc.) in fixed locations or areas of interest are detected, and defects (such as bridging, missing solder, and cold solder joints) are identified. Another example is the automatic creation and categorization of captured photos in a terminal, which can also assist with image editing and text / image retrieval.

[0041] Current research on semantic segmentation mainly focuses on supervised semantic segmentation, which requires supervised training on finely labeled artificial datasets. The trained neural network models are usually only applicable to the categories or scenarios covered by the dataset, and their generalization performance is poor, which is far from meeting the requirements for open-set semantic segmentation.

[0042] To bridge the gap between supervised semantic segmentation and real-world applications, several basic zero-shot semantic segmentation models have been proposed. However, most of these methods explore deep semantic relationships between unseen and seen classes, requiring large amounts of labeled data and a wider variety of seen class categories. The training datasets for these methods are often limited to a single dataset or datasets with specific content, making them prone to overfitting to the corresponding scene or category and unable to effectively segment the diverse categories of the real world. For example, if a user's dataset only contains the animal category as a visible category, existing zero-shot methods cannot detect categories in autonomous driving scenarios, such as bicycles.

[0043] Based on the above analysis, it is clear that how to use a model with open set segmentation capabilities to correctly process images in the above-mentioned application scenarios is a technical problem that has not yet been solved.

[0044] To address the problem of the inability to perform semantic recognition of images, this application employs a visual language-driven language segmentation method. This method uses visual and textual embedding features obtained based on visual and textual comparisons, respectively, to perform semantic segmentation on the image to be predicted. It does not require manually labeled data, thus significantly improving the semantic segmentation effect and enabling rapid semantic recognition of images.

[0045] like Figure 1 The diagram shown illustrates the implementation principle of an image processing method provided in this application. This method achieves semantic segmentation through a semantic segmentation model, which includes a visual encoder and a clustering head.

[0046] The semantic segmentation model receives an image with predictions via a visual encoder; extracts pixel-level embedding features from the image; and sends the pixel-level embedding features to the clustering head.

[0047] The semantic segmentation model performs clustering analysis based on the pixel-level embedding features provided by the visual encoder through a clustering head, and outputs the clustering results.

[0048] The semantic segmentation model uses a visual encoder to pool pixel-level embedding features based on the clustering results to obtain visual embedding features corresponding to multiple image regions.

[0049] The semantic segmentation model can obtain predefined text categories and extract features from these categories to obtain text embedding features. Furthermore, the semantic segmentation model also includes a text encoder, which can receive predefined text categories.

[0050] The semantic segmentation model performs semantic segmentation on multiple image regions based on their respective visual embedding features and text embedding features, in order to obtain the semantic segmentation result for each image region.

[0051] In some embodiments, the semantic segmentation model may specifically be the ViL-Seg model. However, it is not limited to this; the semantic segmentation model provided in this application embodiment may also be other semantic segmentation models, such as U-Net, Fully Convolutional Networks (FCN), (A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, SegNet), DeepLab v1, etc.

[0052] As explained in the inference process of the aforementioned semantic segmentation model, this embodiment employs a visual-language-driven language segmentation method, utilizing image-text pairs for semantic segmentation without requiring any fine-grained annotation. Based on image-text segmentation, this embodiment uses visual contrast and language contrast to obtain fine-grained visual embedding features and coarse-grained text embedding features, respectively. This embodiment uses a clustering head to divide the visual embedding features into different semantic regions. During the inference stage, it also proposes an alignment method based on region pooling, resulting in smoother segmentation results. Furthermore, both training and inference are performed end-to-end, offering the advantages of simplicity and feasibility.

[0053] Based on the aforementioned semantic segmentation model, embodiments of this application provide an image processing method, such as... Figure 2 As shown, this image processing method achieves semantic segmentation through a semantic segmentation model, which includes a visual encoder and a clustering head. The image processing method further includes:

[0054] 201. Input the image to be predicted into the visual encoder to obtain pixel-level embedding features.

[0055] The image to be predicted can be a single image or multiple local images after data augmentation. Furthermore, the image to be predicted can be an RGB image; the specific type and number of images are not limited in this embodiment.

[0056] This visual encoder, also known as an image encoder, can be a visual encoder (VisualTransformer), such as the ViT-B / 16. The visual encoder has visual feature extraction capabilities, extracting pixel-level features from the image to be predicted to obtain pixel-level embedded features.

[0057] 202. Input the pixel-level embedded features into the clustering head to obtain the clustering results.

[0058] The clustering head can receive pixel-level embedding features from the visual encoder, and can perform clustering analysis on these pixel-level embedding features to obtain clustering results.

[0059] In some embodiments of this application, the clustering head can be an online clustering head.

[0060] Among them, the online clustering head can dynamically divide pixel-level embedded features into different semantic regions, and the alignment method based on region pooling during the inference stage makes the semantic segmentation results smoother.

[0061] 203. Pool the pixel-level embedding features based on the clustering results to obtain the visual embedding features corresponding to multiple image regions.

[0062] After obtaining the clustering results output by the clustering head, the visual encoder can perform pooling processing on these results, such as average pooling. Through pooling, visual embedding features corresponding to multiple image regions can be obtained, which can be region image embedding features.

[0063] 204. Based on the visual embedding features and text embedding features corresponding to multiple image regions, perform semantic segmentation on multiple image regions to obtain the semantic segmentation result corresponding to each image region; the text embedding features are obtained based on predefined text categories.

[0064] In this embodiment, a predefined text category can be pre-input into the semantic segmentation model, and text embedding features can be obtained through the predefined text category. For example, the semantic segmentation model obtains visual embedding features corresponding to multiple image regions through a visual encoder, combines them with the pre-obtained text embedding features, and then performs semantic segmentation on the multiple image regions according to the visual embedding features and text embedding features to obtain the semantic segmentation results of the image regions.

[0065] In some embodiments of this application, the semantic segmentation model may further include a text encoder, into which predefined text categories are input to obtain text embedding features.

[0066] The text encoder can receive predefined text categories, also known as semantic categories, and extract text embedding features corresponding to the image through the text encoder.

[0067] In some embodiments of this application, the semantic segmentation model further includes: a text encoder; inputting predefined text categories into the text encoder to obtain text embedding features, including:

[0068] A1. Combine predefined text categories and predefined prompt templates to obtain text;

[0069] A2. Input the text into the text encoder to obtain text embedding features corresponding to multiple text categories.

[0070] The text encoder combines text categories with prompt templates to generate text. This text is then input into the text encoder to obtain text embedding features. Prompt templates, used in natural language research (e.g., "a photo of a {category}"), are used to form sentences resembling natural language, serving as a cross-modal pre-training method. By combining prompt templates and text categories, text can be obtained quickly.

[0071] In this embodiment, unlabeled text categories can be input into the semantic segmentation model, which can reduce the workload of manual annotation.

[0072] In some embodiments of this application, step 204 performs semantic segmentation on multiple image regions based on their respective visual embedding features and text embedding features, to obtain a semantic segmentation result for each image region, including:

[0073] B1. Calculate the similarity between the visual embedding features corresponding to each image region and the text embedding features of multiple text categories to obtain multiple similarity values ​​for each image region.

[0074] B2. The text category corresponding to the highest similarity value among multiple similarity values ​​is taken as the semantic segmentation result for each image region.

[0075] This can be achieved by calculating the similarity between the visual embedding features and text embedding features of different image regions, obtaining multiple similarity values ​​for each image region, and then selecting the maximum similarity value to obtain the semantic segmentation result for each image region. For a detailed explanation of the specific calculation process for the similarity values, please refer to the description in the subsequent embodiments.

[0076] In one possible implementation, the semantic segmentation results can be used in the fields of autonomous driving, product and device defect detection, or terminal image classification and recognition.

[0077] The semantic segmentation results are used to identify non-whitelisted objects on the road surface in autonomous driving systems; or to detect defects in product components; or to classify, edit, and retrieve text-based images from terminal devices. In the above schemes, semantic segmentation can be applied to the needs of open-set semantic segmentation, identifying images in the field of autonomous driving, detecting defects in product components, analyzing images captured by terminal devices, assisting in image editing, and performing text-based image retrieval and analysis.

[0078] As illustrated by the foregoing embodiments, the semantic segmentation model includes a visual encoder and a clustering head. The image to be predicted is input into the visual encoder to obtain pixel-level embedding features; the pixel-level embedding features are input into the clustering head to obtain clustering results; the pixel-level embedding features are pooled based on the clustering results to obtain visual embedding features corresponding to multiple image regions; semantic segmentation is performed on the multiple image regions based on the visual embedding features and text embedding features corresponding to the multiple image regions to obtain semantic segmentation results for each image region; the text embedding features are obtained based on predefined text categories. In this embodiment, a visual-based contrast model is used, which allows the semantic segmentation model to better learn fine-grained visual embedding features. Furthermore, it only requires predefined text categories to encode text embedding features, eliminating the need for manually labeled data. Therefore, it significantly improves the semantic segmentation effect and quickly achieves semantic recognition of images.

[0079] In some embodiments of this application, the image processing method provided in this application, in addition to performing the aforementioned... Figure 2 In addition to the steps shown, this image processing method also includes:

[0080] C1. Train the semantic segmentation model using image-text pairs without labeled text categories, and then output the trained semantic segmentation model.

[0081] First, image-text pairs without labeled text categories are obtained. For example, these image-text pairs can be image-text pair data from the Internet without any fine-grained annotation. In this embodiment, the above-mentioned image-text pairs can be used as training data to train the original semantic segmentation model, thereby completing the semantic segmentation training and outputting the trained semantic segmentation model.

[0082] Furthermore, in some embodiments of this application, the aforementioned step C1 uses image text pairs without labeled text categories to train the semantic segmentation model, including:

[0083] C11. Extract the original image and its corresponding original text from the image-text pair.

[0084] The input to the semantic segmentation model consists of image-text pairs without labeled text categories, which include pairs of original images and original text.

[0085] C12. Perform image data enhancement on the original image to obtain multiple local images corresponding to the original image.

[0086] In this embodiment, image data enhancement is performed on the original image to obtain multiple local images, and the image enhancement strategy used is not limited.

[0087] C13. Input the original image and multiple local images into the visual encoder to obtain global visual embedding features and local visual embedding features.

[0088] The visual encoder takes the original image and multiple local images as input. The semantic segmentation model can extract global visual embedding features and local visual embedding features through the backbone network of the ViT-B / 16 image encoder. These two embedding features extract global and local information of the image, respectively.

[0089] C14. Pool the global visual embedding features and the local visual embedding features respectively to obtain the pooled global visual embedding features and the pooled local visual embedding features.

[0090] The visual encoder performs pooling on both global and local visual embedding features, dividing the original image into multiple image regions to obtain pooled global and local visual embedding features. For example, average pooling can be applied to both global and local visual embedding features to obtain the global image embedding feature.

[0091] C15. Cross-modal contrasting supervision is performed on the original text embedding features and the pooled global visual embedding features to obtain cross-modal contrasting results. The original text embedding features are obtained by extracting features from the original text through a semantic segmentation model.

[0092] For example, the raw text can be input into a text encoder to obtain the raw text embedding features.

[0093] The text encoder extracts the original text to obtain the original text embedding features, which can also be called word embedding features.

[0094] After obtaining the original text embedding features and the pooled global visual embedding features, cross-modal contrastive supervised learning is performed using the original text embedding features and the pooled global visual embedding features. Specifically, a cross-modal contrastive supervised loss function can be defined to obtain the cross-modal contrast results.

[0095] C16. Input the global visual embedding features and local visual embedding features into the clustering head, and calculate the maximum mutual information of the clustering results distribution of the same pixel between the global visual embedding features and local visual embedding features through the clustering head.

[0096] Specifically, the online clustering head can take global visual embedding features and local visual embedding features as input. The online clustering head can calculate the maximum mutual information of the clustering results distribution of the same pixel between the global visual embedding features and the local visual embedding features. The maximization of mutual information of the online clustering head enables the semantic segmentation model to automatically segment the semantic regions of visual features.

[0097] C17. The semantic segmentation model is trained by maximizing the mutual information between the cross-modal contrast results and the clustering results of the same pixel, and the training ends when the semantic segmentation model converges.

[0098] In the aforementioned steps, the maximum mutual information of the cross-modal comparison results and the clustering results of the same pixel is generated respectively. By optimizing the maximum mutual information of the cross-modal comparison results and the clustering results of the same pixel, the semantic segmentation model can be trained. At each optimization, it is determined whether the semantic segmentation model has converged. Finally, the training of the semantic segmentation model can be terminated.

[0099] Furthermore, in some embodiments of this application, step C17 optimizes the mutual information of the cross-modal contrast results and the clustering results of the same pixel to maximize the mutual information, thereby training the semantic segmentation model, and terminates the training when the semantic segmentation model converges, including:

[0100] C171. Perform vision-based contrasting supervision on the pooled global visual embedding features and the pooled local visual embedding features to obtain the visual contrasting results;

[0101] In addition, visual contrast supervision can be introduced into the visual encoder. Specifically, visual contrast supervision can be performed on the pooled global visual embedding features and the pooled local visual embedding features, and visual contrast results can be generated.

[0102] C172. The semantic segmentation model is trained by maximizing the mutual information of visual contrast results, cross-modal contrast results, and clustering results of the same pixel, and training ends when the semantic segmentation model converges.

[0103] In the aforementioned steps, visual contrast supervision can be introduced into the visual encoder to improve the training effect of the semantic segmentation model. During the training phase of the model, the semantic segmentation model can be trained by optimizing the distribution of visual contrast results, cross-modal contrast results, and clustering results of the same pixel to maximize mutual information.

[0104] Furthermore, in some embodiments of this application, the aforementioned step C171 performs visual contrast supervision on the pooled global visual embedding features and the pooled local visual embedding features to obtain visual contrast results, including:

[0105] C1711. Perform fully connected mapping on the pooled global visual embedding features and the pooled local visual embedding features to obtain the fully connected mapping result.

[0106] C1712. Calculate the semantic consistency between global visual embedding features and local visual embedding features based on the fully connected mapping results, and calculate the loss function to obtain the visual contrast results including the loss function.

[0107] The visual encoder can generate a fully connected mapping result based on the pooled global visual embedding features and the pooled local visual embedding features. Based on the semantic consistency between the global visual embedding features and the local visual embedding features, a visual contrast supervision loss function is calculated. For example, cross-entropy can be used as the loss function to obtain the visual contrast result.

[0108] Furthermore, in some embodiments of this application, step C15 above performs cross-modal contrast supervision on the original text embedding features and the pooled global visual embedding features to obtain cross-modal contrast results, including:

[0109] C151. When there are multiple image-text pairs, calculate the cross-modal supervised loss function for the multiple image-text pairs based on the original text embedding features and the pooled global visual embedding features.

[0110] C152. Calculate the average value of the cross-modal supervision loss function for multiple image-text pairs to obtain the cross-modal comparison results.

[0111] The visual encoder can calculate the cross-modal supervision loss function for each image-text pair. The average value of the cross-modal supervision loss functions for multiple image-text pairs can be calculated to obtain the cross-modal comparison result. In this embodiment, cross-modal supervision can align the embedding spaces of visual features and text features.

[0112] To facilitate a better understanding and implementation of the above-described solutions in the embodiments of this application, specific examples of corresponding application scenarios are provided below.

[0113] The application scenarios of this application mainly focus on open-set semantic segmentation tasks on open-world datasets. As a crucial component of the vision field, semantic segmentation has wide applications across various industries, such as autonomous driving, product and component defect detection, and terminal image classification and recognition. Autonomous driving is a complex robotic task that requires perception, planning, and execution in constantly changing environments. Due to its paramount safety requirements, this task must be performed with the highest accuracy. Semantic segmentation provides information about free space on the road, as well as information such as lane markings and traffic signs. In the field of autonomous driving, high-precision segmentation of long-tailed scenes is particularly important. Long-tailed scenes typically contain very complex road conditions, with multiple objects existing in the same scene, and various combinations of these objects, all of which affect segmentation accuracy. Datasets often focus on a few scenes or some common categories (e.g., people or vehicles). Models trained on such datasets can usually only segment the categories defined by the dataset, but cannot segment undefined new categories, posing a significant challenge to the safety of autonomous driving systems. For example, tires and pieces of paper on the road are often not in the categories labeled in existing autonomous driving datasets, making it impossible to detect these obstacles on the road and reducing the safety of existing autonomous driving systems.

[0114] This application embodiment can employ a semantic segmentation model based on image-text comparison and visual comparison in open dataset scenarios, improving the performance and robustness of the semantic segmentation model on data or categories with missing annotations in open datasets. Through image-text comparison, the semantic segmentation model provided in this application embodiment can learn complex semantic information. Simultaneously, leveraging the characteristics of internet data, the semantic segmentation model can segment uncommon categories, achieving good results when applied to various scenarios. For example, the semantic segmentation model provided in this application embodiment accurately segments various uncommon categories based on given nouns, such as masked images of dinosaurs, hot air balloons, or yoga mats, even without having seen these category annotations before.

[0115] This application employs a network model for image-text (VisionLanguage) contrast learning segmentation. It can acquire fine-grained visual feature information of images through visual contrast, thereby obtaining a more accurate segmentation mask. It also acquires coarse-grained semantic information of images through language contrast, thus effectively determining the category of the segmentation mask. Coarse-grained semantics refers to mapping the entire image to category text, which cannot effectively handle pixel-level information. Fine-grained visual features refer to learning details at the image patch or pixel level by aligning the features of the cropped image with those of the original image. Furthermore, this application proposes an online clustering head to better divide visual features into different semantic regions. It should be noted that this application only requires collecting a wide range of naturally existing image-text pairs as the training dataset, without requiring additional manual annotation. Moreover, this application can be applied to semantic segmentation tasks on most image datasets and achieves good results.

[0116] The terminal device provided in this application only needs to be a regular camera capable of acquiring two-dimensional image data, which can be deployed on a corresponding computing device depending on the scene. In subsequent embodiments, the semantic segmentation model is the ViL-Seg model for example. The image processing method flow provided in this application embodiment will be described next.

[0117] First, the embodiments of this application can be divided into two processes: the ViL-Seg model training process and the ViL-Seg model inference process.

[0118] like Figure 3 and Figure 4 As shown, the ViL-Seg model mainly consists of the following three parts: visual encoder, text encoder, and clustering head.

[0119] The visual encoder, also known as the image encoder, can specifically be a VisionTransformer (e.g., ViT-B / 16), used to extract pixel-level visual embedding features and global visual embedding features. In the vision-based contrastive supervision, a multi-cropping strategy is used to crop six local blocks from the image, with a mapping layer dimension of 2048 and a visual embedding feature dimension of 512.

[0120] The text encoder can be a TextTransformer (e.g., ViT-B / 16) used to extract text embedding features corresponding to the image. The text embedding features have a dimension of 512 and can be used to calculate the cosine similarity between the text embedding features and the visual embedding features.

[0121] The clustering head is designed based on the ViL-Seg model. For example, this clustering head can be an online clustering head, and the number of clustering channels C of the online clustering head is 25.

[0122] This application embodiment uses the Adam optimizer, which can be used to train the aforementioned model with a learning rate of 5e. -4 That is, 5*10 -4 It is trained on a dataset of 12 million images and text (e.g., CC12M).

[0123] like Figure 3 The diagram shows the training process of the ViL-Seg model, which mainly consists of three supervision signals: cross-modal supervision, which enables the model to learn coarse-grained semantic category features; visual basis supervision, which enables the model to learn fine-grained visual features; and maximizing the mutual information of the online clustering heads, which allows the model to automatically segment semantic regions from visual features.

[0124] During the model training phase, given an RGB image x as input, the image is first augmented to obtain multiple local images. The ViL-Seg model then extracts global and local visual embedding features through a ViT-B / 16 visual encoder backbone network. These two embedding features extract global and local information from the image, respectively, and are then processed by average pooling to obtain the global image embedding feature. Next, visual-based contrastive supervised learning is performed using the global and local information of the image.

[0125] For the text t corresponding to the RGB image x, the ViL-Seg model obtains text embedding features through a Transformer text encoder backbone network. These text embedding features extract the text information corresponding to the image, including image category information. Cross-modal comparative supervised learning is performed using these text embedding features and global image embedding features. The ViL-Seg model inputs the pixel-level embedding features into the aforementioned clustering head to obtain the probability distribution of each pixel-level embedding feature belonging to each category. Learning is achieved by maximizing the mutual information of image features for the same pixel under different data augmentations. The entire training process is conducted on a large-scale image-text pair dataset without detailed segmentation annotations collected from the internet.

[0126] like Figure 4 The diagram shows the inference process of the ViL-Seg model. The inference process uses the weights of the image encoder and text encoder trained in the training process to encode the image and text features respectively. It uses the trained clustering head to cluster the image features of different regions and calculates the similarity to obtain the optimal match between the image features and text features of different regions.

[0127] like Figure 4As shown, in the inference stage of the ViL-Seg model, this embodiment only requires an image and a predefined category as input to obtain the semantic segmentation result. The ViL-Seg model inputs image data into a trained image encoder to obtain pixel-level embedding features, and obtains clustering results through a clustering head. Then, visual embedding features corresponding to different image regions are obtained through cluster-based pooling.

[0128] The text category is combined with a prompt template to obtain text. This text is then input into a trained text encoder to obtain text embedding features. The prompt template, used in natural language research (e.g., a photo of a {category}), is used to form a natural language-like sentence with the category name and can be used for cross-modal pre-training. The text category corresponding to the largest cosine similarity between the visual embedding features and the text embedding features is the semantic segmentation category for that image region.

[0129] The training and inference processes of the semantic segmentation model will be described in detail below, including the following steps:

[0130] Step 1: The input is an RGB image x. Using a multicrop image enhancement strategy g(·), the RGB image x is transformed into k local images, x local =g(x)=[x l1 x l2 , ..., x lk ], where k is the number of local images.

[0131] Step 2: All images from Step 1, including local and global images, are input into the visual encoder to obtain the global feature e. v (x) and local features e v (x local Local features are the stitching of multiple views [e] v (x l1 ), e v (x l2 ), ..., e v (x lk )).

[0132] Step 3: Apply a fully connected mapping Φ to the features from Step 2. a (e v (x)), Φ a (e v (x local )).

[0133] Step 4: Perform comparative supervision on the features obtained in Step 3 to learn the semantic information consistency between global and local features, using cross-entropy as the loss function L. vision =H(Φ a (e v (x)), Φ a (e v (x local ))).

[0134] Step 5: Input the text t corresponding to the RGB image in Step 1, and process it through a text encoder e. t In this embodiment, a pre-trained Transformer model is used to obtain the text embedding features e. t (t).

[0135] Step 6: The image-text pairs in Step 1 and Step 5 are mini-batch b pairs {x} j , t i} j＝1 b b represents the number of images in the current batch.

[0136] When m = n, the image feature e v (x m ) and text features e t (t n If it is a positive pair, then it is a negative pair. The cross-modal supervision loss function for this mini-batch is:

[0137] Where ⊙ is defined as cosine similarity, for example... τ is defined as a temperature parameter, which is a hyperparameter used to smooth or sharpen the obtained distribution. The final objective function is the average value of the cross-modal supervision loss function in the mini-batch. Where a = e v (x m b = e t (t m Aligning visual embedding features with text embedding features allows the category information in the text to be transferred to the visual embedding space. Therefore, the similarity between visual embedding features and text embedding features of different classes can be compared to obtain image category information.

[0138] Step 7: Obtain the visual embedding features of each pixel in the global and local images based on the visual encoder in Step 3. The feature dimension is HW·D, where H is the height of the image, W is the width of the image, and D is the number of channels.

[0139] Step 8: Construct the cluster head Φ cThe clustering head consists of a C-channel convolutional layer and a loss (softmax) function, which inputs pixel-level visual embedding features into the clustering head.

[0140] Step 9: Define q, q′∈R 1·D It refers to the feature obtained from the same pixel in different data-augmented image data in pixel-level visual embedding features, with the goal of maximizing the mutual information between the two. Φ c These represent the parameters of the online clustering head. q and q′ are different features of the same pixel obtained by the visual encoder in different data-augmented image data.

[0141] Step 10: Optimize the loss function L vision L cross , Until it converges.

[0142] Step 11: In the inference phase of the ViL-Seg model, using the visual encoder, text encoder, and clustering head from Step 10, input the input image x into the visual encoder to obtain pixel-level embedding features. Passing this to the clustering head yields C clustering masks M∈{0,1} HWC .

[0143] Step 12: The C clustering masks generated in Step 11 are filtered and average pooled based on pixel indices to obtain the pooling features of all pixels at the region level. Here, filtering refers to filtering out pixels that are marked as not belonging to this class (marked as 0) by the mask.

[0144] Step 13: The category of each region in Step 12 is obtained by calculating the similarity between the region-level features and the text embedding features of different categories. For words of different categories, the hint "a photo of a[]" can be added to improve the semantic segmentation effect.

[0145] As illustrated by the foregoing examples, this application's embodiments conducted comprehensive and detailed experiments on the ViL-Seg model to demonstrate its effectiveness. These embodiments were tested on the full test sets of the PASCALVOC, PASCALContext, and COCOStuff datasets. Since other methods used manually labeled visible classes for training, this application's embodiments compared the segmentation results on invisible classes with other methods for a fair comparison.

[0146] The PASCAL VOC dataset test set contains 1449 images, of which there are 5 invisible classes: potted plant, sheep, sofa, train, and TV monitor.

[0147] The PASCALContext dataset test set contains 5105 images, of which there are 4 invisible classes: cow, motorbike, sofa, and cat.

[0148] The COCOStuff dataset test set contains 5000 images, of which there are 15 invisible classes: frisbee, skateboard, cardboard, carrot, scissots, suitcase, giraffe, cow, road, wallconcrete, tree, grass, river, clouds, and playingfield.

[0149] In this embodiment, mean Intersection over Union (mIoU) and pixel accuracy (pix.acc.) can be used as evaluation metrics. Table 1 below shows a comparison of semantic segmentation results for different datasets.

[0150]

[0151] Table 1 compares the zero-shot segmentation of invisible classes by different semantic segmentation methods on different datasets. Table 1 shows the comparison between the ViL-Seg model and other methods on the PASCALVOC, PASCAL Context, and COCOStuff datasets ("n / a" indicates that the other methods were not tested for this result). As shown in Table 1, the ViL-Seg model outperforms all other methods in terms of mIoU and pixel accuracy on the three datasets. This demonstrates that the embodiments of this application can learn a segmentation-capable model from image-text pairs on the Internet to segment different open-world categories. ZS3 requires training on datasets with fine-grained annotations, while ViL-Seg does not, yet still outperforms ZS3, which greatly saves on the cost of manual annotation. ViL-Seg's improvement on PASCALVOC is better than the other two datasets, possibly because PASCALVOC has 15 visible classes, while PASCALContext has 55 and COCOStuff has 168.

[0152] In this embodiment, the original data requiring manual annotation is discarded, and the required image-text pairs are directly obtained from the Internet, thus reducing costs. Due to the wide range and scalability of the dataset, this embodiment can be applied to any downstream task and achieve very good results. A visual contrast-based ViL-Seg model is proposed, which can better learn fine-grained visual features and significantly improve the semantic segmentation effect. This embodiment uses an online clustering head, which reduces inference time, improves inference efficiency, and enhances the semantic segmentation effect.

[0153] In other embodiments of this application, the individual effects of each module can be determined, including the impact of simply not using visual contrast-based supervision and using offline clustering on the overall result.

[0154] Using the same dataset settings as in the previous embodiments, the effects of several modules proposed in this application's embodiments were obtained. Ablation experiments were conducted to eliminate the effects based on visual contrast and online clustering (or offline K-means clustering). As follows:

[0155] Table 2 is a comparison table of the effects of ViL-Seg after eliminating several modules.

[0156]

[0157] As shown in Table 2, the ViL-Seg model proposed in this application significantly reduces performance after eliminating visual contrast-based methods. This indicates that in addition to cross-modal supervision to align the embedding spaces of visual and textual data, the visual encoder is also constrained by its own self-supervised signals from local and global image patches. Without visual contrast-based methods, the clustering results cannot distinguish semantically meaningful objects from background regions, and the global signals are insufficient to support fine-grained tasks. On the three datasets, the online clustering method provided in this application has higher segmentation performance than the offline clustering method. It can be considered that the online clustering head is tightly coupled with the visual encoder, and can improve the quality of visual embeddings as training progresses, which is unattainable by offline methods. The frame rate (Frames Per Second, FPS) of the online clustering head is 3.53, while that of the offline clustering is 2.97. The offline clustering is slower because it requires post-processing of neural network features, while the online clustering head in this application improves the inference speed.

[0158] This application proposes a visual language-driven semantic segmentation framework, ViL-Seg, which collects massive amounts of image-text pairs from the internet for model training, improving the generalization ability of the semantic segmentation model. It proposes a scheme combining cross-modal contrastive and visual contrastive approaches, simultaneously ensuring both coarse-grained classification and fine-grained feature segmentation capabilities for different semantic objects. Employing an online clustering head, the semantic segmentation model can be trained and inferred end-to-end, optimizing the quality of segmented images and improving inference speed.

[0159] Not limited to this, the online clustering head in the embodiments of this application enables the semantic segmentation model to improve the quality of segmented images while increasing inference speed. The embodiments of this application can also be extended to other semantic segmentation frameworks or systems, such as at least one of the following: U-Net, FCN, SegNet, DeepLab v1, etc. It can be combined with other visual embeddings to improve the quality of segmentation maps while increasing inference speed.

[0160] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0161] To facilitate better implementation of the above-described solutions in the embodiments of this application, related apparatus for implementing the above-described solutions is also provided below.

[0162] Please see Figure 5 As shown in the embodiment of this application, a terminal device 500 is provided. The semantic segmentation model includes a visual encoder and a clustering head. The terminal device includes a visual encoding module 501, a clustering module 502, a pooling module 503, and a semantic segmentation module 504.

[0163] A visual encoding module is used to input the image to be predicted into the visual encoder to obtain pixel-level embedding features;

[0164] The clustering module is used to input the pixel-level embedded features into the clustering head to obtain the clustering results;

[0165] The pooling module is used to pool the pixel-level embedding features based on the clustering results to obtain visual embedding features corresponding to multiple image regions respectively.

[0166] The semantic segmentation module is used to perform semantic segmentation on the multiple image regions based on the visual embedding features and text embedding features corresponding to the multiple image regions respectively, so as to obtain the semantic segmentation result corresponding to each image region; the text embedding features are obtained based on predefined text categories.

[0167] As illustrated by the foregoing embodiments, the semantic segmentation model includes a visual encoder and a clustering head. The image to be predicted is input into the visual encoder to obtain pixel-level embedding features; the pixel-level embedding features are input into the clustering head to obtain clustering results; the pixel-level embedding features are pooled based on the clustering results to obtain visual embedding features corresponding to multiple image regions; semantic segmentation is performed on the multiple image regions based on the visual embedding features and text embedding features corresponding to the multiple image regions to obtain semantic segmentation results for each image region; the text embedding features are obtained based on predefined text categories. In this embodiment, a visual-based contrast model is used, which allows the semantic segmentation model to better learn fine-grained visual embedding features. Furthermore, it only requires predefined text categories to encode text embedding features, eliminating the need for manually labeled data. Therefore, it significantly improves the semantic segmentation effect and quickly achieves semantic recognition of images.

[0168] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiments of this application, and the resulting technical effects are the same as those of the method embodiments of this application. For details, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.

[0169] This application also provides a computer storage medium storing a program that performs some or all of the steps described in the above method embodiments.

[0170] The following describes another terminal device provided in the embodiments of this application. Please refer to [link / reference]. Figure 6 As shown, the terminal device 600 includes:

[0171] Receiver 601, transmitter 602, processor 603, and memory 604 (wherein the terminal device 600 may contain one or more processors 603). Figure 6 (Taking a processor as an example). In some embodiments of this application, the receiver 601, transmitter 602, processor 603, and memory 604 can be connected via a bus or other means, wherein... Figure 6 Taking the example of a connection between China and Israel via a bus.

[0172] Memory 604 may include read-only memory and random access memory, and provides instructions and data to processor 603. A portion of memory 604 may also include non-volatile random access memory (NVRAM). Memory 604 stores operating systems and operating instructions, executable modules or data structures, or subsets thereof, or extended sets thereof. The operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic business functions and handling hardware-based tasks.

[0173] Processor 603 controls the operation of the terminal device. Processor 603 can also be called a central processing unit (CPU). In specific applications, the various components of the terminal device are coupled together through a bus system. This bus system includes not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0174] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 603. Processor 603 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 603 or by instructions in the form of software. The processor 603 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 604. Processor 603 reads the information in memory 604 and, in conjunction with its hardware, completes the steps of the above method.

[0175] The receiver 601 can be used to receive input digital or character information and generate signal inputs related to the settings and function control of the terminal device. The transmitter 602 may include a display device such as a display screen and can be used to output digital or character information through an external interface.

[0176] In this embodiment, processor 603 is used to execute the aforementioned... Figure 2 Any step in the image processing method shown.

[0177] In another possible design, when the terminal device is specifically a chip, the chip includes a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer-executable instructions stored in a storage unit to cause the chip within the terminal to perform any of the methods described in the first aspect above. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the terminal, such as read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0178] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0179] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0181] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0182] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. An image processing method, characterized by, The semantic segmentation model includes a visual encoder and a clustering head, wherein the method includes: The image to be predicted is input into the visual encoder to obtain pixel-level embedding features; The pixel-level embedding features are input into the clustering head to obtain the clustering results; The pixel-level embedding features are pooled based on the clustering results to obtain visual embedding features corresponding to multiple image regions respectively. Semantic segmentation is performed on the multiple image regions based on their respective visual embedding features and text embedding features to obtain a semantic segmentation result for each image region; the text embedding features are obtained based on predefined text categories. The method further includes: The semantic segmentation model is trained using image-text pairs without labeled text categories, and then the trained semantic segmentation model is output.

2. The method of claim 1, wherein, The semantic segmentation model also includes: a text encoder; The method further includes: The text is obtained by combining predefined text categories and predefined prompt templates. The text is input into the text encoder to obtain text embedding features corresponding to multiple text categories.

3. The method of claim 2, wherein, The step of performing semantic segmentation on the multiple image regions based on their respective visual embedding features and text embedding features to obtain a semantic segmentation result for each image region includes: Calculate the similarity between the visual embedding features corresponding to each image region and the text embedding features of the multiple text categories to obtain multiple similarity values ​​corresponding to each image region; The text category corresponding to the highest similarity value among the multiple similarity values ​​is used as the semantic segmentation result for each image region.

4. The method of claim 1, wherein, The step of training the semantic segmentation model using image-text pairs without labeled text categories includes: Obtain the original image and the corresponding original text from the image-text pair; Image data enhancement is performed on the original image to obtain multiple local images corresponding to the original image; The original image and the multiple local images are input into the visual encoder to obtain global visual embedding features and local visual embedding features; The global visual embedding features and the local visual embedding features are respectively pooled to obtain pooled global visual embedding features and pooled local visual embedding features; Cross-modal comparison supervision is performed on the original text embedding features and the pooled global visual embedding features to obtain cross-modal comparison results. The original text embedding features are obtained by feature extraction of the original text through the semantic segmentation model. The global visual embedding features and the local visual embedding features are input into the clustering head, and the clustering head is used to calculate the maximum mutual information of the clustering results distribution of the same pixel between the global visual embedding features and the local visual embedding features; The semantic segmentation model is trained by maximizing the mutual information between the cross-modal comparison results and the clustering results of the same pixel, and training ends when the semantic segmentation model converges.

5. The method according to claim 4, characterized in that, The step of optimizing the cross-modal contrast results and the clustering results of the same pixel to maximize mutual information to train the semantic segmentation model, and ending training when the semantic segmentation model converges, includes: Visual contrast supervision is performed on the pooled global visual embedding features and the pooled local visual embedding features to obtain visual contrast results; The semantic segmentation model is trained by maximizing the mutual information of the visual contrast results, the cross-modal contrast results, and the clustering results of the same pixel, and training ends when the semantic segmentation model converges.

6. The method according to claim 5, characterized in that, The step of performing visual contrast supervision on the pooled global visual embedding features and the pooled local visual embedding features to obtain visual contrast results includes: A fully connected mapping is performed on the pooled global visual embedding features and the pooled local visual embedding features to obtain the fully connected mapping result; The semantic consistency between global visual embedding features and local visual embedding features is calculated based on the fully connected mapping result, and a loss function is calculated to obtain a visual contrast result including the loss function.

7. The method according to any one of claims 4 to 6, characterized in that, The method of performing cross-modal contrast supervision on the original text embedding features and the pooled global visual embedding features to obtain cross-modal contrast results includes: When there are multiple image-text pairs, the cross-modal supervised loss function of the multiple image-text pairs is calculated based on the original text embedding features and the pooled global visual embedding features; The average value of the cross-modal supervision loss function for the plurality of image-text pairs is calculated to obtain the cross-modal comparison results.

8. The method according to any one of claims 1 to 6, characterized in that, The clustering head includes: an online clustering head.

9. The method according to any one of claims 1 to 6, characterized in that, The semantic segmentation results are used in the fields of autonomous driving, product and component defect detection, or terminal image classification and recognition.

10. A terminal device, comprising: The semantic segmentation model includes a visual encoder and a clustering head, wherein the terminal device includes: A visual encoding module is used to input the image to be predicted into the visual encoder to obtain pixel-level embedding features; The clustering module is used to input the pixel-level embedded features into the clustering head to obtain the clustering results; The pooling module is used to pool the pixel-level embedding features based on the clustering results to obtain visual embedding features corresponding to multiple image regions respectively. The semantic segmentation module is used to perform semantic segmentation on the multiple image regions based on the visual embedding features and text embedding features corresponding to the multiple image regions respectively, so as to obtain the semantic segmentation result corresponding to each image region; the text embedding features are obtained according to predefined text categories; The terminal device is also used to train the semantic segmentation model using image text pairs without labeled text categories, and then output the trained semantic segmentation model.

11. A terminal device, comprising: The terminal device includes: a processor and a memory; the processor and the memory communicate with each other. The memory is used to store instructions; The processor is configured to execute the instructions in the memory, performing the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1-9.

13. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-9.

Images