Image segmentation method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an image segmentation method and device, electronic equipment and storage medium, and relates to the technical field of artificial intelligence. The method comprises: acquiring a support image and a query image, the support image comprising a base class object and a new class object; based on the support image, extracting a first image feature, the first image feature being used for segmenting the base class object in the query image; based on the support image and the query image, extracting a second image feature, the second image feature being used for segmenting the base class object and the new class object in the query image; combining the first image feature and the second image feature to obtain a third image feature; and based on the third image feature, segmenting the query image to obtain a first image segmentation result. The embodiment of the present disclosure can effectively improve the pollution of the new class feature to the base class feature in the new class registration process by combining the original base class feature with the base class feature after the new class registration, so as to make the image segmentation result more accurate.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an image segmentation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of computer science and technology, image processing and analysis have gradually formed their own scientific system. Image segmentation is a crucial preprocessing step in image processing and computer vision; without correct segmentation, correct recognition is impossible. Therefore, people hope to introduce artificial intelligence methods to accurately segment images. While this improves the efficiency and accuracy of image segmentation, it also increases the complexity of solving the problem.

[0003] Image segmentation involves pixel-by-pixel classification, requiring highly labeled data. Therefore, it heavily relies on large datasets with dense segmentation labels, which is extremely time-consuming and labor-intensive. Related techniques often use few-shot segmentation to address this problem. Few-shot segmentation can learn and predict new class features with a small number of samples, provided that sufficient base class features have been learned. However, during prediction, the learning of new class features may influence the base class features, reducing the accuracy of the prediction results. Summary of the Invention

[0004] In view of this, this disclosure relates to the field of artificial intelligence technology, and more specifically, to an image segmentation method, apparatus, computer device, and storage medium.

[0005] In a first aspect, an image segmentation method is provided, comprising: acquiring a support image and a query image, wherein the support image includes base class objects and new class objects; extracting a first image feature based on the support image, wherein the first image feature is used to segment the base class objects in the query image; extracting a second image feature based on the support image and the query image, wherein the second image feature is used to segment the base class objects and new class objects in the query image; combining the first image feature and the second image feature to obtain a third image feature; and segmenting the query image based on the third image feature to obtain a first image segmentation result.

[0006] In some embodiments, after segmenting the query image based on the third image features to obtain a first image segmentation result, the method further includes: adjusting the first image segmentation result based on the similarity between the query image and the supporting images; segmenting the query image based on the first image features to obtain a second image segmentation result; and fusing the adjusted first image segmentation result and the second image segmentation result to obtain a third image segmentation result.

[0007] In some embodiments, adjusting the first image segmentation result based on the similarity between the query image and the supporting image includes: calculating the first gram matrix of the supporting image and the second gram matrix of the query image respectively; using the norm of the difference between the first gram matrix and the second gram matrix as a guiding factor; and adjusting the first image segmentation result based on the guiding factor.

[0008] In some embodiments, after fusing the adjusted first image segmentation result and the second image segmentation result to obtain the third image segmentation result, the method further includes: segmenting the query image based on the spatial location features of the query image to obtain the fourth image segmentation result, wherein the spatial location features indicate the positional relationship between each object to be segmented in the query image, and the objects to be segmented include at least one of base class objects and new class objects; and fusing the fourth image segmentation result and the third image segmentation result to obtain the fifth image segmentation result.

[0009] In some embodiments, fusing the fourth image segmentation result with the third image segmentation result to obtain the fifth image segmentation result includes: performing channel-by-channel stitching of the fourth image segmentation result and the third image segmentation result to obtain the fifth image segmentation result.

[0010] In some embodiments, segmenting the query image based on the spatial location features of the query image to obtain a fourth image segmentation result includes: obtaining the spatial location features of the supporting image; and obtaining the fourth image segmentation result based on the similarity between the spatial location features of the supporting image and the spatial location features of the supporting image of the query image.

[0011] In some embodiments, extracting second image features based on a supporting image and a query image includes: extracting a first new class feature of a new class of objects in the supporting image; predicting the new class of objects in the query image based on the first image feature to obtain a second new class feature; and combining the first new class feature, the second new class feature, and the first image feature to obtain the second image feature.

[0012] In some embodiments, combining a first new class feature, a second new class feature, and a first image feature to obtain a second image feature includes: combining the first new class feature and the first image feature based on a first adaptive weight to obtain a fourth image feature, wherein the first adaptive weight is used to indicate the degree of influence of the first new class feature on the first image feature; combining the second new class feature and the first image feature based on a second adaptive weight to obtain a fifth image feature, wherein the second adaptive weight is used to indicate the degree of influence of the second new class feature on the first image feature; and combining the fourth image feature and the fifth image feature to obtain the second image feature.

[0013] In a second aspect, an image segmentation apparatus is provided, comprising: an acquisition module for acquiring a support image and a query image, the support image including base class objects and new class objects; a first extraction module for extracting a first image feature based on the support image, the first image feature being used to segment the base class objects in the query image; a second extraction module for extracting a second image feature based on the support image and the query image, the second image feature being used to segment the base class objects and new class objects in the query image; a combination module for combining the first image feature and the second image feature to obtain a third image feature; and a segmentation module for segmenting the query image based on the third image feature to obtain a first image segmentation result.

[0014] Thirdly, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of the first aspect by executing the executable instructions.

[0015] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method of the first aspect described above.

[0016] The image segmentation method provided in this disclosure acquires a support image and a query image. Based on the support image and the query image, it extracts a first image feature containing only base class information and a second image feature containing both base class information and new class information. These two features are combined to obtain a third image feature, which is then used to segment the query image. This disclosure effectively reduces the contamination of base class features during the new class registration process by superimposing the original base class features onto the base class features after new class registration, thus resulting in more accurate image segmentation results. Attached Figure Description

[0017] Figure 1 A schematic diagram of the system architecture of an image segmentation method according to an embodiment of this disclosure is shown.

[0018] Figure 2 The diagram shows a flowchart of an image segmentation method according to an embodiment of the present disclosure.

[0019] Figure 3 This diagram illustrates the process after the query image is segmented based on third image features to obtain the first image segmentation result in an embodiment of this disclosure.

[0020] Figure 4 This diagram illustrates the process of obtaining a third image segmentation result from the first and second image segmentation results after fusion and adjustment, according to an embodiment of this disclosure.

[0021] Figure 5 A flowchart illustrating another image segmentation method in an embodiment of this disclosure is shown.

[0022] Figure 6 A schematic diagram of the structure of an image segmentation apparatus according to an embodiment of the present disclosure is shown.

[0023] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0025] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] With the continuous development of computer science and technology, digital images have become an effective tool for researchers in many fields such as military, remote sensing, and meteorology. Therefore, artificial intelligence image processing technology has attracted widespread attention. Image segmentation is a crucial preprocessing step in image processing and computer vision; without correct segmentation, correct recognition is impossible. Image segmentation requires highly labeled data and heavily relies on large datasets with dense segmentation labels, which is extremely time-consuming and labor-intensive. Therefore, few-sample segmentation methods based on a small number of samples have emerged.

[0027] In related technologies, few-shot segmentation can learn and predict new class features with a small number of samples, provided that the base class features have been learned from a sufficient number of samples. Specifically, few-shot segmentation adds a new class registration stage on top of the few-shot learning, resulting in better prediction performance. However, during the new class registration stage, the new class features in the query image and support images alter the original accurate base class features, reducing segmentation accuracy.

[0028] In view of this, the present disclosure provides an image segmentation method. This method involves acquiring a support image and a query image, extracting a first image feature containing only base class information and a second image feature containing both base class information and new class information from the support image and query image, combining the two to obtain a third image feature, and then segmenting the query image based on the third image feature. Therefore, the embodiments of the present disclosure can effectively reduce the contamination of base class features during the new class registration process, thereby making the image segmentation results more accurate.

[0029] This disclosure provides an image segmentation method, apparatus, electronic device, and storage medium. Specifically, the image segmentation apparatus can be integrated into an electronic device, such as a terminal or server.

[0030] It is understood that the image segmentation method of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting this disclosure.

[0031] Figure 1 An exemplary system architecture diagram is shown that can be applied to the image segmentation method or image segmentation apparatus of the present disclosure embodiments.

[0032] like Figure 1 As shown, the system architecture 100 includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a network, such as a wired or wireless network, wherein the image segmentation device can be integrated into the server.

[0033] Server 102 can be used to obtain supporting images and query images. The supporting images include base class objects and new class objects. Based on the supporting images, a first image feature is extracted, which is used to segment the base class objects in the query image. Based on the supporting images and the query image, a second image feature is extracted, which is used to segment the base class objects and new class objects in the query image. The first image feature and the second image feature are combined to obtain a third image feature. Based on the third image feature, the query image is segmented to obtain the first image segmentation result.

[0034] Server 102 can be a single server, a server cluster composed of multiple servers, or a cloud server. For example, a server can be an interoperability server between multiple heterogeneous systems, a backend server, an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc. As in the speech synthesis method or apparatus disclosed in this disclosure, multiple servers can form a blockchain, and the server is a node on the blockchain.

[0035] Terminal 101 can send supporting images and query images to server 102, or receive image segmentation results generated by server 102. Terminal 101 may include a mobile phone, smart TV, tablet computer, laptop computer, or personal computer (PC), etc. A client may also be configured on terminal 101, which may be an application client or a browser client, etc.

[0036] Those skilled in the art will know that Figure 1 The number of terminals and servers shown is merely illustrative. Depending on actual needs, there may be any number of terminals and servers, and this disclosure does not impose any limitation on this.

[0037] The following will describe the exemplary implementation method in detail with reference to the accompanying drawings and embodiments.

[0038] First, this disclosure provides an image segmentation method that can be executed by any electronic device with computing capabilities.

[0039] Figure 2 This diagram illustrates a flowchart of an image segmentation method according to an embodiment of the present disclosure, as shown below. Figure 2 As shown, the image segmentation method provided in this embodiment includes the following steps.

[0040] S201, retrieve supporting images and query images. Supporting images include base class objects and new class objects.

[0041] In some embodiments, the supporting image and query image include one or more images containing different categories of identified objects, such as bicycles, cars, horses, people, cats, and dogs.

[0042] The objects to be identified are divided into base class objects and new class objects. Base class objects are known objects that have sufficient training samples, while new class objects are objects that have only a small number of training samples.

[0043] In related technologies, image segmentation requires highly labeled data and relies on a large number of densely labeled training samples. Since base class objects have sufficient training samples, a high-performing base classifier can be trained based on these samples, accurately segmenting the base class objects contained in the image. However, for new class objects with only a few training samples, classifiers trained using the same training method as for base class objects often fail to achieve good segmentation results. Therefore, the image segmentation method in this embodiment can segment images containing new class objects based on sufficient training samples for base class objects and a small number of training samples for new class objects, achieving good segmentation results.

[0044] For example, in a set of landscape images, the base class objects can be plants, lakes, people, and dogs, and the new class objects can be airplanes and bicycles. Since landscape images typically include plants, lakes, people, and dogs, sufficient learning samples of the base class objects plants, lakes, people, and dogs can be obtained; however, landscape images rarely include airplanes and bicycles, so only a small number of learning samples of the new class objects airplanes and bicycles can be obtained.

[0045] S202, Based on the supporting image, extract the first image features.

[0046] Specifically, the base class cb in the supporting image can be extracted. , i The feature vectors of each class are used to select the feature center point of each class's feature vector as the prototype feature of that base class. The prototype features of all base classes together constitute the first image feature in this disclosed embodiment. Where b represents the base class, i is the index used to distinguish each base class, and cls represents the original classifier.

[0047] In some embodiments, a pre-trained Residual Network (ResNet) can be used to process the supporting image to obtain first image features. The shallow layers of the Residual Network extract local detailed texture features, while the deeper layers extract more abstract information such as contours and sizes.

[0048] S203, based on the supporting image and the query image, extract the second image features.

[0049] Specifically, feature vectors of new class objects can be extracted from both the support image and the query image. These new class prototype features are then combined with the base class prototype features to generate second image features. Since the second image features contain information about both the base class and the new class, both base class and new class objects in an image can be identified simultaneously.

[0050] In some embodiments, feature vectors of the supporting image can be extracted using masked average pooling, and their prototype features can be calculated as the first new class features. It is worth noting, however, that the first new class features contain not only new class information but also base class information.

[0051] The formula for calculating the features of the first new class is as follows:

[0052]

[0053] Wherein, the subscript 'sup' represents the supporting image, indicating that the first new class feature is obtained based on the supporting image, and 'h' and 'w' represent the size of the supporting image. Represents base class c b,i New Class Features The mask (i.e., the label) on the top, n b This indicates the number of base classes in the supporting image that contains the new class. This indicates support for the new class c in the image. n,u The j-th supporting image, ° represents the Hadamard product, n represents the novel, and N n The number of new classes is represented by i and u, which are ordinal numbers used to distinguish between different base classes and new classes, respectively. K represents the number of supported images, and j is an ordinal number used to distinguish between different supported images.

[0054] Based on the first image features, the feature vectors in the query image are predicted, and the feature vectors are classified using a classification model to obtain temporary prototype features of the query image, which serve as the second new class features. Here, qry represents the query image, indicating that the second new class feature is obtained based on the query image. For example, the classification model could be a softmax classifier.

[0055] The first new class feature, the second new class feature, and the first image feature obtained above are combined to obtain the second image feature.

[0056] In some embodiments, a fourth image feature can be obtained by combining a first new class feature and a first image feature using a first adaptive weight, and a fifth image feature can be obtained by combining a second new class feature and a first image feature using a second adaptive weight. The fourth and fifth image features are then combined to obtain a second image feature.

[0057] For example, adaptive weights can be used first. By combining the first new class feature and the first image feature, the fourth image feature is obtained. The specific calculation formula is as follows:

[0058]

[0059] Through adaptive weights By combining the second new class feature and the first image feature, the fifth image feature is obtained. The specific calculation formula is as follows:

[0060]

[0061] Here, * indicates element-wise multiplication. Adaptive weights. and The values ​​n can be pre-defined to represent the degree of influence of the first new type of feature and the second new type of feature on the first image feature, respectively. b and N b These represent the number of base classes in the supporting images containing the new class and the number of base classes in the query image, respectively. The other parameters have the same meaning as before.

[0062] The fourth and fifth image features obtained from the above calculations are added together to obtain the second image features. Its calculation formula is:

[0063] It is worth noting that the second image features contain both base class and new class information, which already achieves good segmentation results. However, during the calculation of the second image features, the new class features, through adaptive weighting, affect the original first image features, contaminating the originally accurate base class features and thus worsening the segmentation effect of the second image features on the base class.

[0064] S204, combine the first image features and the second image features to obtain the third image features.

[0065] To solve the above problems, the first image features and the second image features can be combined. In this way, the base class can be accurately identified based on the first image features, while new classes can be identified based on the second image features.

[0066] In some embodiments, the first image features and the second image features can be added together to obtain the third image features.

[0067] S205, based on the third image features, segment the query image to obtain the first image segmentation result.

[0068] In some embodiments, it can be based on third image features The query image is segmented to obtain the first image segmentation result. This segmentation result is a mask containing category information.

[0069] The third image feature contains both the first and second image features, thus it can correct the base class contamination in the second image feature by using the accurate base class information provided by the first image feature. Therefore, when segmenting query images based on the third image feature, it can accurately identify the base class while recognizing the new class, thereby achieving better recognition results.

[0070] In some embodiments, such as Figure 3 As shown, after segmenting the query image based on the third image features to obtain the first image segmentation result, the following steps are also included.

[0071] S301, Adjust the segmentation result of the first image based on the similarity between the query image and the supporting images.

[0072] Specifically, the relationship between query images and supporting images can be established through similarity metrics.

[0073] In some embodiments, a first gram matrix for the supporting image and a second gram matrix for the query image can be calculated separately. Specifically, the process of calculating the first gram matrix for the supporting image includes:

[0074] By utilizing the feature vectors of the support images extracted from the shallow layers of the ResNet, for example, the feature vector F of the support images extracted from the second layer can be used. layer2 (S). And the first Gram matrix G is calculated. s The calculation process is as follows:

[0075]

[0076] G s =A s A s T

[0077] in, Used to adjust the size of the input feature vector, A s This represents the feature vectors of the supported image after deformation. The Gram matrix reflects the degree of correlation between the individual vectors in the feature vectors.

[0078] Using the same method, the feature vector F of the query image extracted by the second-layer network can be used. layer2 (Q), calculate the second Gram matrix G q .

[0079] The guiding factor ψ is calculated based on the first and second gram matrices, using the following formula:

[0080] ψ=‖G s -G q ||F

[0081] in,‖·‖ F This represents the Frobenius norm.

[0082] The first image segmentation result is adjusted based on the guiding factor, and the adjustment process is as follows: Where r1 represents the first image segmentation result obtained based on the third image features. This represents a convolution operation with initial parameters ψ.

[0083] S302, the query image is segmented based on the first image features to obtain the second image segmentation result.

[0084] Specifically, the query image can be segmented using the first image features to obtain the second image segmentation result r2.

[0085] S303, the adjusted first image segmentation result and the second image segmentation result are fused to obtain the third image segmentation result.

[0086] In some embodiments, fusion can be achieved through convolutional computation. And r2, to obtain the third image segmentation result r3, the fusion process is as follows

[0087] The embodiments of this disclosure can adjust the first image segmentation result based on the guidance factor, which can make full use of the information provided by the shallow image features, making the distance between features of different recognition objects larger, while the distance between features of the same recognition object smaller, thereby enabling more accurate differentiation of different recognition objects and optimizing the segmentation result.

[0088] In some embodiments, such as Figure 4 As shown, after fusing and adjusting the first and second image segmentation results to obtain the third image segmentation result, the following steps are also included.

[0089] S401, the query image is segmented based on its spatial location features to obtain the fourth image segmentation result.

[0090] Specifically, spatial location features refer to the spatial positions or relative orientations between multiple objects in an image. These relationships can be categorized into adjacency, overlap, and containment relationships. Spatial location features can enhance the ability to distinguish objects in an image.

[0091] In some embodiments, the query image can be segmented by extracting the spatial location features of the supporting image and the query image, and based on the similarity between the spatial location features of the supporting image and the query image, to obtain a fourth image segmentation result.

[0092] Specifically, feature vectors of the support images extracted from the deeper layers of the ResNet can be utilized. For example, the feature vector F of the support images extracted from the last layer can be used. final (S) and the feature vector F of the query image final (Q). Extract the spatial location features f from the two feature vectors respectively. s and f q And calculate f s and f q For example, cosine similarity can be used to measure the similarity between two objects:

[0093]

[0094] For each spatial location feature of the query image, there is a corresponding spatial location feature of the supporting image with the highest similarity. Therefore, the query image can be segmented based on the feature vector of the supporting image with the highest similarity to the spatial location feature of the query image, generating a fourth image segmentation result r4.

[0095] S402, the fourth image segmentation result and the third image segmentation result are fused to obtain the fifth image segmentation result.

[0096] In some embodiments, the fourth image segmentation result r4 and the third image segmentation result r3 can be concatenated channel by channel to obtain the fifth image segmentation result r5. The concatenation process is as follows: in This indicates a channel-by-channel concatenation operation. Concatenation represents an expansion in channel depth. For example, two 512-channel segmentation results are concatenated to generate a 1024-channel segmentation result.

[0097] The embodiments of this disclosure can effectively utilize the spatial positional relationships of each identified object in the query image by querying the spatial positional features of the image, thereby promoting the interaction between the query image and the supporting image and further optimizing the segmentation results.

[0098] To facilitate understanding, the following will combine... Figure 5 This illustrates another image segmentation method according to embodiments of the present disclosure. For example... Figure 5 As shown, the image segmentation method includes the following steps:

[0099] S501, retrieve supporting images, query images.

[0100] Specifically, the query image is at least one image to be segmented.

[0101] S502, Extract image features.

[0102] Specifically, a convolutional neural network is used to extract image features from the supporting image and the query image to obtain base class features, new class features of the supporting image, and new class features of the query image, which are respectively the first image features, the first new class features, and the second new class features.

[0103] S503, generate the fourth image feature.

[0104] Specifically, the new class information is registered to the first image feature using the first new class feature to obtain the fourth image feature;

[0105] S504, generate the fifth image feature.

[0106] Specifically, the new class information is registered to the first image feature using the second new class feature to obtain the fifth image feature.

[0107] S505, generate the third image feature.

[0108] Specifically, the first image feature, the fourth image feature, and the fifth image feature are combined to obtain the third image feature.

[0109] S506, segment the query image based on the first image features and the third image features.

[0110] The query image is segmented based on the first image feature and the third image feature respectively, resulting in the first and second image segmentation results.

[0111] S507, Fusing image segmentation results.

[0112] Specifically, the first and second image segmentation results are fused based on the guidance factor to obtain the third image segmentation result.

[0113] S508, based on spatial location feature optimization, yields the final segmentation result.

[0114] Specifically, the segmentation result of the third image is optimized based on the spatial location features of the query image and the supporting images to obtain the final segmentation result.

[0115] Based on the same inventive concept, this disclosure also provides an image segmentation apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the method embodiments described above, the implementation of this apparatus embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be repeated.

[0116] Figure 6 This diagram illustrates the structure of an image segmentation apparatus according to an embodiment of the present disclosure, such as... Figure 5 As shown, the image segmentation device 600 includes: an acquisition module 601, a first extraction module 602, a second extraction module 603, a combination module 604, and a segmentation module 605.

[0117] Specifically, the acquisition module 601 is used to acquire a supporting image and a query image, wherein the supporting image includes base class objects and new class objects. The first extraction module 602 is used to extract a first image feature based on the supporting image, and the first image feature is used to segment the base class objects in the query image. The second extraction module 603 is used to extract a second image feature based on the supporting image and the query image, and the second image feature is used to segment the base class objects and new class objects in the query image. The combination module 604 is used to combine the first image feature and the second image feature to obtain a third image feature. The segmentation module 605 is used to segment the query image based on the third image feature to obtain a first image segmentation result.

[0118] In some embodiments, the segmentation module 605 is further configured to: adjust the first image segmentation result based on the similarity between the query image and the supporting image; segment the query image based on the first image features to obtain a second image segmentation result; and fuse the adjusted first image segmentation result and the second image segmentation result to obtain a third image segmentation result.

[0119] In some embodiments, the segmentation module 605 is further configured to: calculate a first gram matrix of the supporting image and a second gram matrix of the query image respectively; use the norm of the difference between the first gram matrix and the second gram matrix as a guiding factor; and adjust the first image segmentation result based on the guiding factor. In some embodiments, the segmentation module 605 is further configured to: segment the query image based on the spatial location features of the query image to obtain a fourth image segmentation result, wherein the spatial location features indicate the positional relationship between each object to be segmented in the query image, and the objects to be segmented include at least one of base class objects and new class objects; and fuse the fourth image segmentation result and the third image segmentation result to obtain a fifth image segmentation result.

[0120] In some embodiments, the segmentation module 605 is further configured to perform channel-by-channel stitching of the fourth image segmentation result and the third image segmentation result to obtain the fifth image segmentation result.

[0121] In some embodiments, the segmentation module 605 is further configured to obtain the spatial location features of the supporting image; and obtain a fourth image segmentation result based on the similarity between the spatial location features of the supporting image and the spatial location features of the supporting image of the query image.

[0122] In some embodiments, the first extraction module 602 is further configured to: extract a first new class feature of a new class of objects in a supporting image; predict the new class of objects in the query image based on the first image feature to obtain a second new class feature; and combine the first new class feature, the second new class feature and the first image feature to obtain a second image feature.

[0123] In some embodiments, the second extraction module 603 is further configured to: combine the first new class feature and the first image feature based on the first adaptive weight to obtain a fourth image feature, wherein the first adaptive weight is used to indicate the degree of influence of the first new class feature on the first image feature; combine the second new class feature and the first image feature based on the second adaptive weight to obtain a fifth image feature, wherein the second adaptive weight is used to indicate the degree of influence of the second new class feature on the first image feature; and combine the fourth image feature and the fifth image feature to obtain a second image feature.

[0124] It should be noted that the image segmentation apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when used for image segmentation. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the image segmentation apparatus and the image segmentation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0125] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0126] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0127] like Figure 7 As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including storage unit 720 and processing unit 710).

[0128] The storage unit stores program code, which can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0129] In some embodiments, the processing unit 710 may perform the following steps of the above method embodiment: acquiring a supporting image and a query image, wherein the supporting image includes base class objects and new class objects; extracting a first image feature based on the supporting image, wherein the first image feature is used to segment base class objects in the query image; extracting a second image feature based on the supporting image and the query image, wherein the second image feature is used to segment base class objects and new class objects in the query image; combining the first image feature and the second image feature to obtain a third image feature; and segmenting the query image based on the third image feature to obtain a first image segmentation result.

[0130] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache memory unit 7202, and may further include a read-only memory unit (ROM) 7203.

[0131] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0132] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0133] Electronic device 700 can also communicate with one or more external devices 740 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0135] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0136] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0138] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0139] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0140] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0141] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0142] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0143] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. An image segmentation method, characterized in that, include: Obtain supporting images and query images, wherein the supporting images include base class objects and new class objects; Based on the supporting image, a first image feature is extracted, which is used to segment the base class object in the query image; Extract the first new class feature of the new class object in the supporting image; Based on the first image features, new class objects in the query image are predicted to obtain the second new class features; Based on the first adaptive weight, the first new class feature and the first image feature are combined to obtain the fourth image feature. The first adaptive weight is used to show the degree of influence of the first new class feature on the first image feature. The fifth image feature is obtained by combining the second new class feature and the first image feature based on the second adaptive weight. The second adaptive weight is used to show the degree of influence of the second new class feature on the first image feature. The fourth image feature and the fifth image feature are combined to obtain the second image feature, which is used to segment the base class object and the new class object in the query image; The first image feature and the second image feature are combined to obtain the third image feature; Based on the third image feature, the query image is segmented to obtain the first image segmentation result.

2. The method according to claim 1, characterized in that, After segmenting the query image based on the third image features to obtain the first image segmentation result, the process further includes: The segmentation result of the first image is adjusted based on the similarity between the query image and the supporting images; The query image is segmented based on the first image features to obtain a second image segmentation result; The first image segmentation result and the second image segmentation result are fused and adjusted to obtain the third image segmentation result.

3. The method according to claim 2, characterized in that, The step of adjusting the first image segmentation result based on the similarity between the query image and the supporting images includes: Calculate the first gram matrix of the supporting image and the second gram matrix of the query image, respectively; The norm of the difference between the first and second Gram matrices is used as a guiding factor. The first image segmentation result is adjusted based on the guidance factor.

4. The method according to claim 2, characterized in that, After fusing and adjusting the first image segmentation result and the second image segmentation result to obtain the third image segmentation result, the method further includes: The query image is segmented based on its spatial location features to obtain a fourth image segmentation result. The spatial location features indicate the positional relationship between each object to be segmented in the query image. The objects to be segmented include at least one of a base class object and a new class object. The fourth image segmentation result and the third image segmentation result are fused to obtain the fifth image segmentation result.

5. The method according to claim 4, characterized in that, The step of fusing the fourth image segmentation result with the third image segmentation result to obtain the fifth image segmentation result includes: The fourth image segmentation result and the third image segmentation result are concatenated channel by channel to obtain the fifth image segmentation result.

6. The method according to claim 4, characterized in that, The segmentation of the query image based on its spatial location features to obtain a fourth image segmentation result includes: Obtain the spatial location features of the supporting image; Based on the similarity between the spatial location features of the supporting image and the spatial location features of the supporting image of the query image, a fourth image segmentation result is obtained.

7. An image segmentation apparatus, characterized in that, include: The acquisition module is used to acquire supporting images and query images, wherein the supporting images include base class objects and new class objects; The first extraction module is used to extract a first image feature based on the supporting image, and the first image feature is used to segment the base class object in the query image; The second extraction module is used to extract a first new class feature of new objects in the supporting image; predict new objects in the query image based on the first image feature to obtain a second new class feature; combine the first new class feature and the first image feature based on a first adaptive weight to obtain a fourth image feature, wherein the first adaptive weight is used to indicate the degree of influence of the first new class feature on the first image feature; combine the second new class feature and the first image feature based on a second adaptive weight to obtain a fifth image feature, wherein the second adaptive weight is used to indicate the degree of influence of the second new class feature on the first image feature; combine the fourth image feature and the fifth image feature to obtain a second image feature, wherein the second image feature is used to segment base class objects and new class objects in the query image; The combining module is used to combine the first image feature and the second image feature to obtain the third image feature; The segmentation module is used to segment the query image based on the third image features to obtain a first image segmentation result.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the image segmentation method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image segmentation method according to any one of claims 1 to 6.

Images