Image segmentation method, image segmentation device and electronic equipment

Abstract

The invention relates to the technical field of computer vision, and discloses an image segmentation method, an image segmentation device and electronic equipment. The image segmentation method comprises the following steps: inputting an image to be segmented into a pre-trained mask generator to obtain a plurality of types-independent masks and mask type probabilities; calculating a centroid point coordinate of each mask, and encoding the centroid point coordinate into a point embedding vector; splicing the point embedding vector and an image block embedding vector of the to-be-segmented image to determine a point feature, and inputting the point feature into a classification head of a ViT-P model to obtain a point category probability; and carrying out weighted fusion on the point category probability and the mask category probability according to a preset weight to obtain an image segmentation result. The category accuracy of the image segmentation result can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, such as an image segmentation method, an image segmentation apparatus, and an electronic device. Background Technology

[0002] Currently, image segmentation can segment multiple objects or parts of an input image by their edges and provide category information, achieving pixel-level understanding of the input image. It is widely used in fields such as autonomous driving, medical image analysis, video surveillance, remote sensing image interpretation, and augmented reality.

[0003] In related technologies, the Transformer architecture or a dual-branch visual Transformer architecture is usually adopted to perform object query prediction masks and categories on the image to be segmented, and output image segmentation results of semantic, instance and panoramic segmentation.

[0004] In the process of implementing the embodiments of this disclosure, it was found that the related technology has at least the following problems: The solutions of related technologies are all based on the Vision Transformer (ViT) visual model to directly encode the features of the image blocks of the image to be segmented, ignoring the features of the mask region in the image, resulting in low category accuracy of the obtained image segmentation results.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0007] One objective of this application is to provide an image segmentation method to address the problem that related technologies, based on the Transformer visual model, directly encode the features of image blocks in the image to be segmented, ignoring the features of the mask region in the image, resulting in low category accuracy of the obtained image segmentation results; a second objective is to provide an image segmentation apparatus; a third objective is to provide another image segmentation apparatus; and a fourth objective is to provide an electronic device.

[0008] In some embodiments, the image segmentation method includes: inputting the image to be segmented into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities; calculating the centroid coordinates of each mask and encoding the centroid coordinates into a point embedding vector; concatenating the point embedding vector with the image patch embedding vector of the image to be segmented to determine point features, and inputting the point features into the classification head of the ViT-P model to obtain point class probabilities; and weighting and fusing the point class probabilities and mask class probabilities according to preset weights to obtain the image segmentation result.

[0009] In this embodiment, when segmenting the image to be segmented, the image is first input into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities. Then, the centroid coordinates of each mask are calculated and encoded as a point embedding vector. This point embedding vector is concatenated with the image patch embedding vector of the image to be segmented to determine the point features. These point features are then input into the classification head of the ViT-P model to obtain the point class probabilities. Finally, the point class probabilities and mask class probabilities are weighted and fused according to preset weights to obtain the final image segmentation result. This achieves complementarity between image patch features and mask point features, reducing the misclassification rate of the segmentation result and improving the class accuracy of the image segmentation result.

[0010] Optionally, the centroid coordinates (x, y) of each mask are calculated according to the following expression. i y i ): ; in, Let (h, w) be the pixel coordinates on the i-th mask, h be the row coordinates of the pixel, and w be a list of pixels. h,w This is to perform a summation operation on the coordinates (h, w) of all pixels on the mask.

[0011] Optionally, encoding the centroid coordinates into a point embedding vector includes: normalizing the centroid coordinates; and mapping the normalized centroid coordinates into a point embedding vector through a linear layer of the point encoder.

[0012] In this embodiment, by normalizing the centroid coordinates, the coordinate dimensions of centroids of masks with different sizes are unified, avoiding the impact of differences in mask area size on subsequent processing. By utilizing the linear layer mapping of the point encoder, low-dimensional coordinates are converted into high-dimensional point embedding vectors, fully exploring the potential features of the centroid coordinates.

[0013] Optionally, concatenating the point embedding vector with the image patch embedding vector of the image to be segmented to determine the point features includes: dividing the image to be segmented into multiple image patches according to a set pixel size, and encoding the multiple image patches to obtain image patch embedding vectors; concatenating them in the order of point embedding vectors first and image patch embedding vectors last to generate an embedding feature sequence; adding the sine position encoding corresponding to the image patch embedding vectors to the embedding feature sequence, and inputting the embedding feature sequence after adding the sine position encoding to a standard Transformer encoder to obtain point features.

[0014] In this embodiment, when determining point features, the image to be segmented is first divided into image patches and encoded into image patch embedding vectors, and local visual features are extracted. Then, the point embedding vectors and image patch embedding vectors are concatenated to form an embedding feature sequence, simultaneously fusing positional and visual feature information. Finally, the embedding feature sequence with added sine wave positions is input to a standard Transformer encoder to be converted into point features. This ensures that the point features simultaneously contain both image patch features and mask point features, improving the accuracy of point feature determination.

[0015] Optionally, the ViT-P model is trained as follows: training sample data labeled using a hybrid annotation strategy is obtained; wherein, the training sample data includes sample data with fine annotation, coarse annotation, and bounding box annotation; the initial model is trained using the training sample data to obtain the ViT-P model.

[0016] In this embodiment, the ViT-P model is obtained by training the initial model with training sample data labeled by a hybrid labeling strategy. This can effectively save labeling costs, shorten the training time of the model, and improve the performance and robustness of the ViT-P model.

[0017] Optionally, before inputting the image to be segmented into a pre-trained mask generator, the image segmentation method further includes scaling the image to be segmented to a set size.

[0018] In this embodiment, the image to be segmented is scaled to a set size before being input into the pre-trained mask generator. This ensures that the input images are of uniform size and guarantees the stability of the mask generator's operation.

[0019] Optionally, after obtaining the image segmentation result, the image segmentation method further includes: determining the masks with confidence scores less than a set threshold in the image segmentation result as noise masks.

[0020] In this embodiment, after obtaining the image segmentation result, masks with a confidence level less than a set threshold are identified as noise masks and removed from the image segmentation result. This removes erroneous parts from the image segmentation result, making subsequent segmentation of the image to be segmented based on the image segmentation result more accurate and reliable.

[0021] In some embodiments, the image segmentation apparatus includes: a classification module for inputting the image to be segmented into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities; an encoding module for calculating the centroid coordinates of each mask and encoding the centroid coordinates into a point embedding vector; a concatenation module for concatenating the point embedding vector with the image patch embedding vector of the image to be segmented to determine point features, and inputting the point features into the classification head of the ViT-P model to obtain point class probabilities; and a weighting module for weighting and fusing the point class probabilities and mask class probabilities according to preset weights to obtain the image segmentation result.

[0022] In some embodiments, the image segmentation apparatus includes a processor and a memory storing program instructions, the processor being configured to execute the image segmentation method as described above when the program instructions are executed.

[0023] In some embodiments, the electronic device includes: a device body; and an image segmentation apparatus as described above, disposed on the device body.

[0024] The beneficial effects of this application are: This application, when segmenting an image, first inputs the image to be segmented into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities. Then, it calculates the centroid coordinates of each mask, encodes them as point embedding vectors, and concatenates these vectors with the image patch embedding vectors of the image to be segmented to determine point features. These point features are then input into the classification head of the ViT-P model to obtain point class probabilities. Finally, the point class probabilities and mask class probabilities are weighted and fused according to preset weights to obtain the final image segmentation result. This approach achieves complementarity between image patch features and mask point features, reducing the misclassification rate of the segmentation result and improving the class accuracy of the image segmentation result.

[0025] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0026] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1This is a schematic diagram of an image segmentation method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of another image segmentation method provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of another image segmentation method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of another image segmentation method provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of an image segmentation apparatus provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of another image segmentation apparatus provided in an embodiment of this disclosure. Detailed Implementation

[0027] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0028] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0029] Unless otherwise stated, the term "multiple" means two or more.

[0030] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0031] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0032] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in the present disclosure can be combined with each other.

[0034] Combination Figure 1 As shown, this disclosure provides an image segmentation method, the execution subject of which can be a processor of an electronic device, and the image segmentation method includes: S101, the processor inputs the image to be segmented into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities.

[0035] Specifically, the image to be segmented refers to the original image that needs to be segmented. This image contains information about various objects or regions, which may belong to different categories.

[0036] Specifically, the mask generator is a model pre-trained on a large amount of image data. It is used to generate a mask from the input image to be segmented. The mask is a binary image where different regions, such as foreground or background, are labeled with different values.

[0037] Specifically, mask category indifference means that when dividing the image by mask, only the division of different regions in the image to be segmented is considered, without regard to the specific category to which these regions belong.

[0038] Specifically, after the image to be segmented is input into the pre-trained mask generator, the mask generator will also output the class probability corresponding to each mask. Here, the class is a classification of the mask itself, such as the classification of features like the shape, size, and position of the mask.

[0039] Optionally, the mask generator is OneFormer.

[0040] For example, with y∈R N×(K+1) This represents a mask, where N is the number of masks, K is the number of categories, and 1 represents the "no category" label.

[0041] S102, the processor calculates the centroid coordinates of each mask and encodes the centroid coordinates into a point embedding vector.

[0042] Specifically, the centroid is the geometric center of an object or region. For each mask, the centroid coordinates of the region corresponding to the mask can be determined by calculating the average of the pixel coordinates of the region. For example, for a region consisting of multiple pixels, the x-coordinate of the centroid is obtained by adding the x-coordinates of all pixels and dividing by the total number of pixels; the y-coordinate is obtained similarly.

[0043] Optionally, the centroid coordinates (x, y) of each mask are calculated according to the following expression. i y i ): ; in, Let (h, w) be the pixel coordinates on the i-th mask, h be the row coordinates of the pixel, and w be a list of pixels. h,w This is to perform a summation operation on the coordinates (h, w) of all pixels on the mask.

[0044] Specifically, a point embedding vector is a method of converting the coordinates of a centroid point into a fixed-dimensional vector. Through a predefined encoding method, such as linear transformation or neural network encoding, the two-dimensional coordinates of the centroid point are transformed into a high-dimensional vector.

[0045] Specifically, by encoding the centroid coordinates into a point embedding vector, the position information of the centroid can be preserved, which facilitates processing in subsequent steps.

[0046] S103, the processor concatenates the point embedding vector with the image patch embedding vector of the image to be segmented to determine the point features, and inputs the point features into the classification head of the ViT-P model to obtain the point class probability.

[0047] Specifically, in image processing, to facilitate model processing, the image is divided into multiple small image patches. The image patch embedding vector is a fixed-dimensional vector transformed from each image patch through an embedding operation. The image patch embedding vector can capture local visual feature information within the image patch.

[0048] Specifically, by concatenating the point embedding vector with the image patch embedding vector of the image to be segmented, the obtained point features can contain both the positional information of the mask centroid and the visual feature information of the local region of the image, enabling the point features to more comprehensively describe the mask-related regions in the image. Therefore, it is necessary to concatenate the point embedding vector with the image patch embedding vector of the image to be segmented to determine the point features.

[0049] Alternatively, a vector concatenation method can be used to concatenate the point embedding vector with the image patch embedding vector of the image to be segmented.

[0050] Specifically, the ViT-P (Vision Transformer-Point) model is a vision model based on the Transformer architecture used to process point features. The classification head is a part of the ViT-P model used to classify and predict the input features. The concatenated point features are input into the classification head of the ViT-P model, and the ViT-P model outputs the class probability corresponding to each point feature, i.e., the point class probability, based on pre-learned features and classification rules.

[0051] S104, the processor performs weighted fusion of point class probabilities and mask class probabilities according to preset weights to obtain image segmentation results.

[0052] Specifically, in the weighted fusion process, the weights of point class probabilities and mask class probabilities need to be pre-set. These pre-set weights can be adjusted based on factors such as the actual application scenario and model performance. For example, when the point class probability more accurately reflects the class information of regions in the image, a larger weight can be assigned to the point class probability. Similarly, when the mask class probability more accurately reflects the class information of regions in the image, a larger weight can be assigned to the mask class probability.

[0053] Specifically, the point class probability and the mask class probability are multiplied by their respective preset weights, and then the two results are added together to obtain the fused probability value, which is the weighted fusion result of the point class probability and the mask class probability.

[0054] Specifically, the point category probability and the mask category probability are weighted and fused according to preset weights. This fusion comprehensively considers information from both point features and mask features, and can more accurately reflect the probability that each region in the image to be segmented belongs to a different category. Therefore, this result can be determined as the image segmentation result.

[0055] For example, the point class probability and the mask class probability are weighted and fused according to a preset weight using the following expression: ; in, For the weighted fusion result, C m Let C be the mask category probability. p Let α be the probability of the point class, and α be the preset weight.

[0056] In this embodiment, when segmenting an image, the image to be segmented is first input into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities. Then, the centroid coordinates of each mask are calculated and encoded as a point embedding vector. This point embedding vector is concatenated with the image patch embedding vector of the image to be segmented to determine the point features. These point features are then input into the classification head of the ViT-P model to obtain the point class probabilities. Finally, the point class probabilities and mask class probabilities are weighted and fused according to preset weights to obtain the final image segmentation result. This achieves complementarity between image patch features and mask point features, reducing the misclassification rate of the segmentation result and improving the class accuracy of the image segmentation result.

[0057] In some embodiments, encoding the centroid coordinates into a point embedding vector includes: normalizing the centroid coordinates; and mapping the normalized centroid coordinates into a point embedding vector through a linear layer of a point encoder.

[0058] It's understandable that the sizes of the masks may differ, resulting in different ranges for the centroid coordinates. Directly using these raw coordinates would lead to significant numerical differences in the centroid coordinates between different masks. Normalization can unify the centroid coordinates to a specific range, such as [0, 1] or [-1, 1], eliminating the dimensional differences caused by varying mask sizes.

[0059] Specifically, a point encoder is a neural network module specifically designed to process point features. It transforms low-dimensional point coordinate information into high-dimensional point embedding vectors to better capture the complex relationships and semantic information between point features. The linear layer is one of the most fundamental layers in a neural network module, performing feature mapping through linear transformations of the input data. Therefore, the linear layer of a point encoder can map the normalized centroid coordinates into point embedding vectors.

[0060] For example, the normalized centroid coordinates are used as... For example, the point embedding vector mapped by the linear layer of the point encoder is: Where N is the same as the number of image blocks, and D is the number of features for embedding encoding, which is usually 768.

[0061] In this embodiment, by normalizing the centroid coordinates, the coordinate dimensions of centroids of masks with different sizes are unified, avoiding the impact of differences in mask area size on subsequent processing. By utilizing the linear layer mapping of the point encoder, low-dimensional coordinates are converted into high-dimensional point embedding vectors, fully exploring the potential features of the centroid coordinates.

[0062] Combination Figure 2 As shown, this disclosure provides another image segmentation method, which includes: S201, the processor inputs the image to be segmented into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities.

[0063] S202, the processor calculates the centroid coordinates of each mask and encodes the centroid coordinates into a point embedding vector.

[0064] S203, the processor divides the image to be segmented into multiple image blocks according to the set pixel size, and encodes the multiple image blocks to obtain image block embedding vectors.

[0065] Specifically, the image to be segmented contains a large amount of pixel information. Directly processing the entire image would result in high computational complexity and difficulty in capturing local features. Therefore, it is necessary to first divide the image to be segmented into multiple image blocks according to a set pixel size.

[0066] Optionally, the pixel size can be set to 16×16 or 32×32.

[0067] Specifically, an image patch is a two-dimensional pixel matrix. By converting the image patch into a fixed-dimensional vector representation, i.e., an image patch embedding vector, it is possible to capture the visual feature information in the image patch.

[0068] For example, using image patches as For example, by encoding image patches, an image patch embedding vector can be obtained. Where C is the number of image features, N' is the number of image blocks, and D is the number of features for block coding.

[0069] S204, the processor concatenates the image features in the order of point embedding vector first and image patch embedding vector last to generate an embedding feature sequence.

[0070] Specifically, the point embedding vector contains the location information of the centroid, while the image patch embedding vector contains the local visual feature information of the image. By concatenating the point embedding vectors first and the image patch embedding vectors last, an embedding feature sequence that simultaneously contains location information and visual features can be generated, facilitating comprehensive processing and analysis in subsequent steps.

[0071] For example, the embedded feature sequence is represented by the following expression:

[0072] Where z is the embedded feature sequence, x p Let x be a point embedding vector. I This is the embedding vector for the image patch.

[0073] S205, the processor adds the sinusoidal position encoding corresponding to the image patch embedding vector to the embedded feature sequence, and inputs the embedded feature sequence after adding the sinusoidal position to the standard Transformer encoder to obtain point features.

[0074] Specifically, the standard Transformer encoder itself lacks the ability to process sequence order information, while features at different locations in an image are crucial for segmentation tasks. Sine positional encoding can add positional information to each element in the embedded feature sequence, enabling the standard Transformer model to distinguish between image patches and point features at different locations. Therefore, it is necessary to add sinusoidal positional encoding corresponding to the image patch embedding vectors to the embedded feature sequence.

[0075] Specifically, the embedded feature sequence is taken as input and passed sequentially through multiple self-attention layers and feedforward neural network layers of a standard Transformer encoder. In each self-attention layer, the standard Transformer encoder calculates the attention weights between each element in the embedded feature sequence and other elements, and then performs a weighted summation of the features based on these weights to update the feature representation of each element. After multiple layers of processing, point features containing rich semantic and spatial information are obtained.

[0076] For example, with Represents point features.

[0077] S206, the processor inputs the point features into the classification head of the ViT-P model to obtain the point class probability.

[0078] S207, the processor performs weighted fusion of point class probabilities and mask class probabilities according to preset weights to obtain image segmentation results.

[0079] In this embodiment, when determining point features, the image to be segmented is first divided into image patches and encoded into image patch embedding vectors, and local visual features are extracted. Then, the point embedding vectors and image patch embedding vectors are concatenated to form an embedding feature sequence, simultaneously fusing positional and visual feature information. Finally, the embedding feature sequence with added sine wave positions is input to a standard Transformer encoder to be converted into point features. This ensures that the point features simultaneously contain both image patch features and mask point features, improving the accuracy of point feature determination.

[0080] In some embodiments, the ViT-P model is trained as follows: training sample data labeled using a hybrid annotation strategy is obtained; wherein the training sample data includes sample data with fine annotation, coarse annotation, and bounding box annotation; the initial model is trained using the training sample data to obtain the ViT-P model.

[0081] Specifically, fine-grained annotation involves sampling centroids and labeling categories from the actual mask, which offers the highest accuracy and can significantly improve mask classification performance.

[0082] Specifically, coarse labeling refers to assigning the same category to randomly sampled points in the approximate boundary region.

[0083] Specifically, the sample data for bounding box annotations are randomly sampled points within the box, which accelerates pre-training, ignores some spatial accuracy but can accelerate the model's effective learning of mask classification capabilities.

[0084] In this embodiment, the ViT-P model is obtained by training the initial model with training sample data labeled by a hybrid labeling strategy. This can effectively save labeling costs, shorten the training time of the model, and improve the performance and robustness of the ViT-P model.

[0085] Combination Figure 3 As shown, this disclosure provides another image segmentation method, which includes: S301, the processor scales the image to be segmented to a set size.

[0086] Optionally, the size can be set to 518×518.

[0087] S302, the processor inputs the image to be segmented into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities.

[0088] S303, the processor calculates the centroid coordinates of each mask and encodes the centroid coordinates into a point embedding vector.

[0089] S304, the processor concatenates the point embedding vector with the image patch embedding vector of the image to be segmented to determine the point features, and inputs the point features into the classification head of the ViT-P model to obtain the point class probability.

[0090] S305, the processor performs weighted fusion of point class probabilities and mask class probabilities according to preset weights to obtain image segmentation results.

[0091] In this embodiment, the image to be segmented is scaled to a set size before being input into the pre-trained mask generator. This ensures that the input images are of uniform size and guarantees the stability of the mask generator's operation.

[0092] Combination Figure 4 As shown, this disclosure provides another image segmentation method, which includes: S401, the processor inputs the image to be segmented into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities.

[0093] S402, the processor calculates the centroid coordinates of each mask and encodes the centroid coordinates into a point embedding vector.

[0094] S403, the processor concatenates the point embedding vector with the image patch embedding vector of the image to be segmented to determine the point features, and inputs the point features into the classification head of the ViT-P model to obtain the point class probability.

[0095] S404, the processor performs weighted fusion of point class probabilities and mask class probabilities according to preset weights to obtain image segmentation results.

[0096] S405, the processor identifies masks with confidence levels less than a set threshold in the image segmentation results as noise masks.

[0097] S406, the processor removes the noise mask from the image segmentation result.

[0098] Specifically, each mask in the image segmentation result corresponds to a confidence level, which reflects the reliability of the segmentation result. If the confidence level of a mask in the image segmentation result is less than a set threshold, it indicates that the reliability of the segmentation result of that mask is low, and it is likely an incorrect segmentation caused by noise interference. Therefore, masks with confidence levels less than the set threshold in the image segmentation result can be identified as noise masks and removed from the image segmentation result.

[0099] In this embodiment, after obtaining the image segmentation result, masks with a confidence level less than a set threshold are identified as noise masks and removed from the image segmentation result. This removes erroneous parts from the image segmentation result, making subsequent segmentation of the image to be segmented based on the image segmentation result more accurate and reliable.

[0100] Combination Figure 5 As shown, this embodiment of the present disclosure provides an image segmentation apparatus 500, including: a classification module 501, an encoding module 502, a stitching module 503, and a weighting module 504. The classification module 501 is used to input the image to be segmented into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities. The encoding module 502 is used to calculate the centroid coordinates of each mask and encode the centroid coordinates into a point embedding vector. The stitching module 503 is used to stitch the point embedding vector with the image patch embedding vector of the image to be segmented to determine point features, and input the point features into the classification head of the ViT-P model to obtain the point class probability. The weighting module 504 is used to weight and fuse the point class probability and the mask class probability according to preset weights to obtain the image segmentation result.

[0101] Combination Figure 6 As shown, this embodiment of the disclosure provides an image segmentation apparatus 600: a processor 601 and a memory 602. Optionally, the apparatus may further include a communication interface 603 and a bus 604. The processor 601, communication interface 603, and memory 602 can communicate with each other via the bus 604. The communication interface 603 can be used for information transmission. The processor 601 can call logical instructions in the memory 602 to execute the image segmentation method of the above embodiment.

[0102] Furthermore, the logical instructions in the aforementioned memory 602 can be implemented as software functional modules and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0103] The memory 602, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 601 executes functional applications and data processing by running the program instructions / modules stored in the memory 602, thereby implementing the image segmentation method in the above embodiments.

[0104] The memory 602 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 602 may include high-speed random access memory and may also include non-volatile memory.

[0105] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to perform the image segmentation method described above.

[0106] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0107] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. An image segmentation method, characterized in that, include: The image to be segmented is input into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities; Calculate the centroid coordinates of each mask and encode the centroid coordinates into a point embedding vector; The point embedding vector is concatenated with the image patch embedding vector of the image to be segmented to determine the point features, and the point features are input into the classification head of the ViT-P model to obtain the point class probability; The point category probability and the mask category probability are weighted and fused according to a preset weight to obtain the image segmentation result.

2. The image segmentation method according to claim 1, characterized in that, Calculate the centroid coordinates (x, y) of each mask using the following expression. i y i ): ； in, Let (h, w) be the pixel coordinates on the i-th mask, h be the row coordinates of the pixel, and w be a list of pixels. h,w This is to perform a summation operation on the coordinates (h, w) of all pixels on the mask.

3. The image segmentation method according to claim 1, characterized in that, Encode the centroid coordinates into a point embedding vector, including: Normalize the coordinates of the centroid point; The normalized centroid coordinates are mapped to point embedding vectors through the linear layer of the point encoder.

4. The image segmentation method according to claim 1, characterized in that, Point features are determined by concatenating the point embedding vector with the image patch embedding vector of the image to be segmented, including: The image to be segmented is divided into multiple image blocks according to a set pixel size, and the multiple image blocks are encoded to obtain image block embedding vectors; The embedding feature sequence is generated by concatenating the point embedding vectors first and the image patch embedding vectors last. Add the sinusoidal position encoding corresponding to the image patch embedding vector to the embedded feature sequence, and input the embedded feature sequence after adding the sinusoidal position to the standard Transformer encoder to obtain point features.

5. The image segmentation method according to claim 1, characterized in that, Train the ViT-P model as follows: Obtain training sample data annotated using a hybrid annotation strategy; the training sample data includes sample data with fine annotation, coarse annotation, and bounding box annotation; The initial model was trained using training sample data to obtain the ViT-P model.

6. The image segmentation method according to any one of claims 1 to 5, characterized in that, Image segmentation methods also include the following steps before inputting the image to be segmented into a pre-trained mask generator: Scale the image to be segmented to the set size.

7. The image segmentation method according to any one of claims 1 to 5, characterized in that, After obtaining the image segmentation results, the image segmentation method also includes: Masks with confidence scores below a set threshold in the image segmentation results are identified as noise masks. Remove the noise mask from the image segmentation results.

8. An image segmentation apparatus, characterized in that, include: The classification module is used to input the image to be segmented into a pre-trained mask generator to obtain multiple class-independent masks and mask class probabilities. The encoding module is used to calculate the centroid coordinates of each mask and encode the centroid coordinates into a point embedding vector; The concatenation module is used to concatenate the point embedding vector with the image patch embedding vector of the image to be segmented to determine the point features, and input the point features into the classification head of the ViT-P model to obtain the point class probability; The weighting module is used to weight and fuse the point category probability and the mask category probability according to preset weights to obtain the image segmentation result.

9. An image segmentation apparatus, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute the image segmentation method as described in any one of claims 1 to 7 when running the program instructions.

10. An electronic device, characterized in that, include: Equipment body; The image segmentation apparatus as described in claim 8 or 9 is disposed on the device body.

Images