Image component segmentation method and system based on dense semantic similarity

Abstract

The present invention discloses a method and system for image component segmentation based on dense semantic similarity, comprising the following steps: collecting image data, using the collected image data as data of a basic image data set; using a target frame to select an object in a target image; extracting image features of the target image and the basic image data set respectively, obtaining global features and descriptors of the image; calculating the dense semantic feature similarity between the global features of each target image and the global features of all basic images, obtaining the most similar basic image; calculating the dense semantic feature similarity between the target image descriptor and the basic image descriptor of the most similar basic image, obtaining the similarity of components between the basic image object and the target object, and segmenting the components of the target object according to the similarity. The present invention performs component segmentation on unseen objects, improves the effect of semantic segmentation of object components, reduces the demand for data sets, and has a wide range of application value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image component segmentation method and system based on dense semantic similarity. Background Art

[0002] With the development of Internet technology and the popularization of intelligent digital products, the number of images in daily life and production has increased dramatically. People need to analyze the constantly emerging images to meet the needs of real life. In robot control, image editing, and image generation applications, it is not enough to simply perceive the object level, but also to segment the parts of the object. Component segmentation is a further analysis of the object, which helps to improve the robot's environmental perception ability and promote the precision processing of images.

[0003] The ability to segment components depends on the number of images and types of components in the training set. However, the number of images in the currently available component datasets is insufficient and the types are single. The number and types of objects and components in the real world far exceed the currently available data, which greatly limits the ability of the component segmentation model to segment components of objects that have not been seen in the training set. Therefore, with the development of artificial intelligence technology in recent years, some new possibilities have been provided for the segmentation of components of unseen objects. We look forward to an image component segmentation method based on dense semantic similarity.

[0004] In target capture, image editing, and image generation applications, it is not enough to only perceive the object level, but also to identify the parts of the target. At present, the types and number of component segmentation datasets are limited, which restricts the development of component segmentation technology. Compared with component-level data, the amount and types of object-level data are rich, so most of the existing technologies are studied under object-level tasks. Component segmentation is not only related to the amount and type of component annotation data, but also closely related to the object category to which the component belongs. That is, the same annotated component has different shapes and colors on different objects. For example, the component annotation is an ear, but the shape and color of a cat's ear and a rabbit's ear are obviously different, which makes component segmentation more challenging. In the component segmentation task, most of them rely on supervised learning methods, but the dataset severely constrains the segmentation effect of this method. Later, with the development of multimodal models, text features are used to assist images in component segmentation. This method greatly improves the effect of component segmentation to a certain extent, but it requires an increase in the calculation of text features. In addition, some studies have performed component segmentation by constructing pseudo-labels and combining them with multimodality. This method has a good effect on segmenting component categories that appear in the dataset, but has a poor effect on processing categories that have not been seen in the dataset.

[0005] The existing technology uses two schemes (one-stage and two-stage) for part semantic segmentation (OV-PARTS: Towards Open-Vocabulary Part Segmentation). The two-stage scheme uses the ZSeg+ model, and the one-stage scheme uses the CLIPSeg model. Both models need to be combined with text features. The ZSeg+ model needs to generate a region of interest first, and then use text features and the region of interest for classification and masking. The CLIPSeg model uses an image encoder to generate image features, a text encoder to generate text features, fuses text and image features, and uses a decoder to generate image masks and categories. Both of the above methods require text features and have higher requirements for data sets. When performing part segmentation of unseen objects, it depends on the quality of the model and text features, but the number and types of part data sets are limited, so the generalization ability of the model trained based on the part data set is weak. In view of the problems in the existing technology that the part data set covers a small number of object types, the number of images with part masks is insufficient, and the cost of part annotation is high, an image part segmentation method based on dense semantic similarity is proposed to perform part segmentation of unseen objects, which is not limited by the number of images and object categories in the data set. In the task of segmenting unseen objects, the existing technology often uses open vocabulary technology for semantic segmentation. However, the dataset used by this method requires not only a large number of images with component masks but also corresponding text annotations. The same object can be described in different texts, which further increases the difficulty of this method. To solve the above problems, this method uses dense semantic similarity features of images to segment components of unseen objects. Summary of the invention

[0006] The purpose of the present invention is to propose an image component segmentation method and system based on dense semantic similarity, which uses the dense semantic similarity features of the image to perform component segmentation of unseen objects. The method of the present invention can complete the component segmentation of objects that do not exist in the basic image dataset without adding text features and additional datasets, providing technical support for target capture, image editing, image generation, etc.

[0007] The present invention is achieved by at least one of the following technical solutions.

[0008] A method for image component segmentation based on dense semantic similarity comprises the following steps:

[0009] Step 1: Collect image data, and use the collected image data as data of the basic image data set;

[0010] Step 2: Use the target frame to select the object in the target image;

[0011] Step 3: extract the image features of the target image and the basic image data set respectively, and obtain the global features of the target image, the global features of the basic image, the target image descriptor, and the basic image descriptor;

[0012] Step 4: Calculate the dense semantic feature similarity between the global features of each target image and the global features of all basic images, and obtain the basic image with the highest similarity, that is, the most similar basic image;

[0013] Step 5: Calculate the similarity of dense semantic features between the target image descriptor and the base image descriptor of the most similar base image, obtain the similarity of components between the base image object and the target object, and segment the components of the target object according to the similarity.

[0014] Furthermore, in step 2, if the target image contains multiple objects, one of the objects is first selected as the target object using the target frame, and the background area outside the target frame is preprocessed;

[0015] The preprocessing includes reducing the background brightness, blurring the background using a Gaussian filter, and processing the objects in the image by cropping;

[0016] Furthermore, the image of the basic image dataset contains only one object, and the object category of the target image is different from the object category of the image of the basic image dataset.

[0017] Furthermore, in step three, the descriptor extraction module uses a VIT model to extract image features.

[0018] Furthermore, the VIT model includes 12 computing unit blocks, and the parameters of the VIT model are adjusted using the image data of the basic image data set: the first 11 computing unit blocks of the VIT model are frozen, and the parameters of the 12th computing unit block are adjusted using the image data of the basic image data set.

[0019] Furthermore, in step 4, The global features of the target image and the Dense semantic feature similarity of global features of base images The formula is as follows:

[0020] ;

[0021] in and They are The global features of the target image and the The global features of the base images, , is a negative number, Represents the length of the vector; the more similar the target image is to the base image, the closer the global feature dense semantic similarity is to 1;

[0022] The first The target image is localized to the most similar base image in the base image dataset:

[0023] ;

[0024] in is the index of the base image that is closest to the global features of the target image. After obtaining the index, the position of the most similar base image in the base image dataset is located. represents the index of the base image dataset, It is The dense semantic feature similarity between the global features of a target image and all global features in the base dataset.

[0025] Furthermore, in step five, The dense semantic feature similarity between the target image descriptor and the base image descriptor of the most similar base image :

[0026] ;

[0027] in, Indicates target image, Represents the index of the most similar base image in the base dataset, and represents the most similar base image obtained in the base image search module, , is a negative number, represents the basic image descriptor obtained by the descriptor extraction module, 、 are the grid coordinates of the underlying image descriptor features, represents the target image descriptor obtained by the descriptor extraction module, 、 is the grid coordinate of the target image descriptor feature, Indicates the length of the vector; when the feature points of the target image descriptor are more similar to the feature points of the base image descriptor, the dense semantic similarity of the descriptor is closer to 1;

[0028] The coordinates of the feature points of the base image descriptor corresponding to each feature point in the target image descriptor and the most similar base image are as follows:

[0029] ;

[0030] in , The coordinates of the feature points of the target image descriptor corresponding to the feature points of the base image descriptor of the most similar base image, 、 are the grid coordinates of the underlying image descriptor features, Represents the dense semantic feature similarity between the target image descriptor and the base image descriptor of the most similar base image.

[0031] Furthermore, in step 5, the category of the component and the position of the component feature point in the target image descriptor are obtained through the feature index of the basic image descriptor, thereby obtaining the component mask of the target object:

[0032] ;

[0033] in, Indicates that the coordinates in the target image are The category of the mask corresponding to the feature, Indicates that the coordinates in the most similar base image are The annotation of the mask corresponding to the feature.

[0034] A system for implementing the image component segmentation method based on dense semantic similarity includes: a target object extraction module, a descriptor extraction module, a basic image search module, and a component semantic segmentation module;

[0035] The target object extraction module obtains the target object through the target detection model;

[0036] The descriptor extraction module extracts the descriptors and global features of the target image and the basic image dataset image through the VIT model;

[0037] The basic image search module is used to locate the most similar basic image between the target image and the basic image data set, and obtain the basic image with the highest similarity, that is, the most similar basic image, by calculating the dense semantic feature similarity between the global features of each target image and the global features of all basic images;

[0038] The component semantic segmentation module is used to segment the components of the object in the target image, and obtains the association between the target object components and the object components of the base image by calculating the dense semantic feature similarity between the target image descriptor and the most similar base image descriptor.

[0039] A computer device comprises: a memory and a processor and a computer program stored in the memory. When the computer program is executed on the processor, the image component segmentation method based on dense semantic similarity is implemented.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention firstly utilizes the VIT model to extract the global features and descriptors of the image; secondly, the dense semantic similarity is calculated between the global features of the target image and the global features of each basic image in the basic image data set, the basic image most similar to the global features of the target image is located according to the dense semantic similarity, and the descriptor of the most similar basic image is extracted; finally, the dense feature similarity between the descriptor of the basic image and the descriptor of the target image is calculated, the component pixels of the target image most similar to the components of the basic image are located according to the dense feature similarity, and the component semantic segmentation module is used to complete the component segmentation of the target image.

[0042] The image component segmentation method based on dense semantic similarity of the present invention can realize component analysis of objects that have not been seen in the basic image data set, thereby reducing the workload of image data annotation; the present invention can save costs and reduce the complexity of segmenting components of objects that have not been seen in the basic image data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of an image component segmentation method based on dense semantic similarity is provided for implementing the present invention;

[0044] Figure 2 A flowchart of a calculation unit block of VIT in the description sub-extraction module in the embodiment;

[0045] Figure 3 A schematic diagram of a component segmentation system based on dense semantic similarity according to an embodiment;

[0046] Figure 4 It is a schematic diagram of dividing the leopard parts by using the present invention. DETAILED DESCRIPTION

[0047] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] like Figure 3 As shown, a component segmentation system based on dense semantic similarity of this embodiment includes: a target object extraction module, a descriptor extraction module, a basic image search module, and a component semantic segmentation module;

[0049] The target object extraction module obtains the target object through a target detection model.

[0050] The descriptor extraction module extracts descriptors and global features of the target image and the basic image dataset image through the VIT model.

[0051] The basic image search module is used to locate the most similar basic image between the target image and the basic image data set, and obtain the basic image with the highest similarity, that is, the most similar basic image, by calculating the dense semantic feature similarity between the global features of the target image and the global features of each basic image.

[0052] The component semantic segmentation module is used to segment (parse) the components of the object in the target image, and obtain the association between the target object components and the object components of the base image by calculating the dense semantic feature similarity between the target image descriptor and the most similar base image descriptor.

[0053] like Figure 1 As shown, a method for segmenting image components based on dense semantic similarity in this embodiment includes the following steps:

[0054] Step 1: Collect image data, and use the collected image data as the data of the basic image data set (training set);

[0055] The images in the basic image dataset contain only one object, and the parts of the object are as rich as possible. Some images can be selected from public part datasets (PASCAL PART, ADE, PartImageNet) as images of the basic image dataset.

[0056] Step 2: Capture the target image and use the target frame to select the object in the target image;

[0057] The object in the target image may be one object or multiple objects. If the target image contains multiple objects, a target frame is first used to select one of the objects as the target object, and the background area outside the target frame is preprocessed.

[0058] The preprocessing includes reducing the background brightness, blurring the background using a Gaussian filter, and processing the objects in the image by cropping;

[0059] The target image is an object image that is not in the basic image data set, and the object category of the target image is different from the image object category of the basic image data set.

[0060] As an embodiment, the target detection model may adopt YoloDETR or the like.

[0061] Step 3: Use the descriptor extraction module to extract the image features of the target image and the basic image data set respectively, and obtain the global features of the target image, the global features of the basic image, the target image descriptor, and the basic image descriptor.

[0062] As an embodiment, the descriptor extraction module uses a VIT (Vision Transformer) model to extract image features. The VIT model includes 12 computing unit blocks, and the image data of the basic image data set is used to adjust the parameters of the VIT model: the first 11 computing unit blocks of the VIT model are frozen, and the image data of the basic image data set is used to adjust the parameters of the 12th computing unit block.

[0063] Before the descriptor extraction module extracts image features, the image is first resized to a uniform size, such as the height of the image. Width is 224 224 or 384 384, etc. The image size in this embodiment is fixed to 224 224 pictures, split the pictures into 14 14 size 16 16 patches are used to generate 196 tokens by flattening them.

[0064] A learnable token cls token is added before the 196 tokens, and a position code is added to the 196 tokens to form a first token; the first token outputs the global features of the target image or the global features of the base image after passing through a calculation unit block (Block).

[0065] Each computational unit block includes a normalization layer, a multi-head attention mechanism, and a feedforward neural network, such as Figure 2 As shown, first, the first token is normalized through a normalization layer to obtain a second token, ensuring that the data is more evenly distributed in different dimensions.

[0066] Secondly, the second token is divided into multiple parts and sent to the multi-head attention mechanism to capture the relationship between tokens to form a third token, and the third token is added to the first token for cross-level feature fusion.

[0067] Again, the fused features are input into the normalization layer for normalization to obtain the fourth token, ensuring that the data continues to be evenly distributed after the multi-head attention mechanism.

[0068] Finally, the fourth token is input into the feedforward neural network, and the fifth token is obtained by processing and conversion through the feedforward neural network. The fifth token is added to the third token for cross-level feature fusion.

[0069] The feedforward neural network includes two fully connected layers and one activation layer.

[0070] The fused features output by the previous computational unit block are used as the input of the next computational unit block. After 12 computational unit blocks, each token is fully processed and transformed to capture the rich feature information of the image. The key of the multi-head attention mechanism in the last computational unit block outputs the target image descriptor or the base image descriptor.

[0071] Step 4: Use the basic image search module to calculate the dense semantic feature similarity between the global features of each target image and the global features of all basic images, and obtain the basic image with the highest similarity, that is, the most similar basic image, including the following steps:

[0072] Dense semantic feature similarity The formula is as follows:

[0073] ;

[0074] in and They are The global features of the target image and the The global features of the base images, , Take -6, To avoid division by zero for smaller values, Represents the length of the vector; the more similar the target image is to the base image, the closer the global feature dense semantic similarity is to 1.

[0075] Locate the base image that is most similar to the target image in the base image dataset:

[0076] ;

[0077] in is the index of the base image that is closest to the global features of the target image. After obtaining the index, the position of the most similar base image in the base image dataset can be located. Represents the index of the underlying image dataset.

[0078] Locate the position of the most similar basic image in the basic image dataset and obtain the file name and file location of the most similar basic image. Use the most similar basic image as the input of the descriptor extraction module, extract the spatial features of the object in the most similar basic image, and extract the key as the basic image descriptor in the most similar basic image in the multi-head attention mechanism in the 12th computational unit block.

[0079] Step 5: Use the component semantic segmentation module to calculate the dense semantic feature similarity between the target image descriptor and the base image descriptor of the most similar base image, obtain the similarity of the components between the base image object and the target object, and segment the components of the target object according to the similarity, including the following steps:

[0080] (1) Similarity of dense semantic features between the target image descriptor and the base image descriptor of the most similar base image:

[0081] ;

[0082] in, Indicates target image, represents the index of the most similar base image in the base dataset, represents the most similar base image obtained in the base image search module, To avoid division by zero for smaller values, represents the basic image descriptor obtained by the descriptor extraction module, 、 are the grid coordinates of the underlying image descriptor features, represents the target image descriptor obtained by the descriptor extraction module, 、 is the grid coordinate of the target image descriptor feature, Represents the length of the vector; when the feature points of the target image descriptor are more similar to the feature points of the base image descriptor, the dense semantic similarity of the descriptor is closer to 1.

[0083] (2) Descriptor dense semantic similarity represents the similarity between each feature point in the target image descriptor and each feature point in the base image descriptor of the most similar base image. Take the first feature point of the target image descriptor and the feature point with the highest degree of similarity among all the feature points of the base image descriptor of the most similar base image, take the second feature point of the target image descriptor and the feature point with the highest degree of similarity among all the feature points of the base image descriptor of the most similar base image, and so on. Each feature point of the target image descriptor has a corresponding feature point of the base image descriptor. The coordinates of the feature point of the base image descriptor corresponding to each feature point in the target image descriptor can be obtained. The formula for this process is as follows:

[0084] ;

[0085] in , Indicates the coordinates of the feature point with the highest similarity to the base image descriptor corresponding to the feature point of the target image descriptor. 、 The grid coordinates of the underlying image descriptor features, Represents the dense semantic feature similarity between the target image descriptor and the base image descriptor of the most similar base image.

[0086] (3) Generate the component mask of the target object. The component mask of the base image is known. Similarly, whether the feature points of the base image descriptor contain components is also known. In addition, the coordinates of the feature points of the base image descriptor with the highest similarity to the feature points of the target image descriptor are known. Therefore, the category of the component and the position of the component feature point in the target image descriptor are obtained through the feature index of the base image descriptor, thereby obtaining the component mask of the target object. The formula of this process is as follows:

[0087] ;

[0088] in, The coordinates in the target image are The category of the mask corresponding to the feature, Indicates that the coordinates in the most similar base image are The category of the mask corresponding to the feature.

[0089] As a specific embodiment, Figure 4 As shown in A and B, the target image is a leopard, the most similar base image is a cat, and the component segmentation diagram of the target image is as follows Figure 4 As shown in C, first, through the dense semantic feature similarity between the global features of the target image (leopard) and the global features of all basic images, the leopard is found to be most similar to the cat in the basic image data set. The dense semantic feature similarity between the descriptor of the leopard and the descriptor of the cat in the most similar basic image is used to obtain the leopard's component segmentation map. For generating the component mask of the target object, different color masks can be used to represent the parts of the cat, such as the cat's eyes are represented by green and yellow masks, the cat's neck is represented by a pink mask, the cat's nose is represented by an orange mask, the cat's trunk is represented by pink-orange, the cat's ears are represented by light blue and blue-indigo, and the cat's head is represented by blue. The component mask of the leopard is obtained by the method of the present invention, the yellow and green masks represent the leopard's eyes, the pink mask represents the leopard's neck, the orange mask represents the leopard's nose, the pink-orange mask represents the leopard's body, the light blue and blue-indigo represent the leopard's ears, and the blue represents the leopard's head.

[0090] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well.

Claims

1. A method for image component segmentation based on dense semantic similarity, characterized in that: The following steps are involved: Step 1: Collect image data, and use the collected image data as data of the basic image data set; Step 2: Use the target frame to select the object in the target image; Step 3: extract the image features of the target image and the basic image data set respectively, and obtain the global features of the target image, the global features of the basic image, the target image descriptor, and the basic image descriptor; Step 4: Calculate the dense semantic feature similarity between the global features of each target image and the global features of all basic images, and obtain the basic image with the highest similarity, that is, the most similar basic image; Step 5: Calculate the similarity of dense semantic features between the target image descriptor and the base image descriptor of the most similar base image, obtain the similarity of components between the base image object and the target object, and segment the components of the target object according to the similarity; Among them The dense semantic feature similarity between the target image descriptor and the base image descriptor of the most similar base image : ； in, Indicates target image, Represents the index of the most similar base image in the base dataset, and represents the most similar base image obtained in the base image search module, , is a negative number, represents the most similar basic image descriptor obtained by the descriptor extraction module, 、 is the grid coordinate of the most similar base image descriptor feature, represents the target image descriptor obtained by the descriptor extraction module, 、 is the grid coordinate of the target image descriptor feature, Indicates the length of the vector; when the feature points of the target image descriptor are more similar to the feature points of the base image descriptor, the dense semantic similarity of the descriptor is closer to 1; The coordinates of the feature points of the base image descriptor corresponding to each feature point in the target image descriptor and the most similar base image are as follows: ； in , The coordinates of the feature points of the target image descriptor corresponding to the feature points of the base image descriptor of the most similar base image, 、 is the grid coordinate of the most similar base image descriptor feature, Represents the dense semantic feature similarity between the target image descriptor and the base image descriptor of the most similar base image.

2. The image component segmentation method based on dense semantic similarity according to claim 1, characterized in that: In step 2, if the target image contains multiple objects, first use the target box to select one of the objects as the target object, and pre-process the background area outside the target box; The preprocessing includes reducing background brightness, blurring the background using a Gaussian filter, and processing the objects in the image by cropping.

3. The image component segmentation method based on dense semantic similarity according to claim 1, characterized in that: The images in the basic image dataset contain only one object, and the object category of the target image is different from that of the image in the basic image dataset.

4. The image component segmentation method based on dense semantic similarity according to claim 1, characterized in that: In step three, the descriptor extraction module uses the VIT model to extract image features.

5. The image component segmentation method based on dense semantic similarity according to claim 4, characterized in that: The VIT model includes 12 computing unit blocks, and the image data of the basic image data set is used to adjust the parameters of the VIT model: the first 11 computing unit blocks of the VIT model are frozen, and the image data of the basic image data set is used to adjust the parameters of the 12th computing unit block.

6. The image component segmentation method based on dense semantic similarity according to claim 1, characterized in that: In step 4, The global features of the target image and the The dense semantic feature similarity of the global features of the base images The formula is as follows: ； in and They are The global features of the target image and the The global features of the base images, , is a negative number, Represents the length of the vector; the more similar the target image is to the base image, the closer the global feature dense semantic similarity is to 1; The first The target image is localized to the most similar base image in the base image dataset: ； in is the index of the base image that is closest to the global features of the target image. After obtaining the index, the position of the most similar base image in the base image dataset is located. represents the index of the base image dataset, It is The dense semantic feature similarity between the global features of a target image and all global features in the base dataset.

7. The image component segmentation method based on dense semantic similarity according to claim 1, characterized in that: In step 5, the category of the component and the location of the component feature points in the target image descriptor are obtained through the feature index of the basic image descriptor, thereby obtaining the component mask of the target object: ； in, Indicates that the coordinates in the target image are The category of the mask corresponding to the feature, Indicates that the coordinates in the most similar base image are The annotation of the mask corresponding to the feature.

8. A system for implementing the image component segmentation method based on dense semantic similarity as described in any one of claims 1 to 7, characterized in that: include: Target object extraction module, descriptor extraction module, basic image search module, component semantic segmentation module; The target object extraction module obtains the target object through the target detection model; The descriptor extraction module extracts the descriptors and global features of the target image and the basic image dataset image through the VIT model; The basic image search module is used to locate the most similar basic image between the target image and the basic image in the basic image data set, and obtain the basic image with the highest similarity, that is, the most similar basic image, by calculating the dense semantic feature similarity between the global features of each target image and the global features of all basic images; The component semantic segmentation module is used to segment the components of the object in the target image, and obtains the association between the target object components and the object components of the base image by calculating the dense semantic feature similarity between the target image descriptor and the most similar base image descriptor.

9. A computer device, characterized in that: The invention comprises: a memory and a processor and a computer program stored in the memory, and when the computer program is executed on the processor, an image component segmentation method based on dense semantic similarity as described in any one of claims 1 to 7 is implemented.

Images