Semantic segmentation method, device, electronic device and storage medium

By introducing a contrast learning and clustering mechanism in the semantic segmentation network, the problem that the new category target in semantic segmentation in small sample is misjudged as background is solved, the semantic segmentation accuracy and training accuracy are improved, and system bias is reduced.

CN114821045BActive Publication Date: 2024-11-15TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210294719.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-23
Publication Date
2024-11-15
Estimated Expiration
2042-03-23

AI Technical Summary

Technical Problem

The prior art tends to use the target of the new category as the background in small sample semantic segmentation, resulting in a reduced semantic segmentation accuracy and a system bias problem during training.

Method used

By introducing the first branch network and the second branch network into the semantic segmentation network, it is used to compare the features of the learning sample support image and query image, and cluster the background features of the query image, enriching the method of feature comparison learning and avoiding the Take the new category goals as background.

Benefits of technology

The semantic segmentation accuracy in small sample semantic segmentation scenarios is improved, the system bias problem is slowed down, and the network training accuracy is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114821045B_ABST
    Figure CN114821045B_ABST
Patent Text Reader

Abstract

The present application provides a semantic segmentation method, device, electronic device and storage medium, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, assisted driving, industrial defect quality inspection, etc. The method includes: obtaining a support image and a query image to be segmented; inputting the support image and the query image into a semantic segmentation network to obtain a semantic segmentation result of the query image; the semantic segmentation network is obtained by training a first branch network and a second branch network in a preset semantic segmentation network based on a small number of sample support images annotated with a first sample category annotation result and a sample query image annotated with a second sample category annotation result. The embodiment of the present application obtains a semantic segmentation network based on the first branch network and the second branch network for training, but the first branch network can be used to improve the semantic segmentation accuracy of the query image during the test process, and since the second branch network is not required, the consumption of additional system resources is not increased.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a semantic segmentation method, device, electronic device and storage medium. Background Art

[0002] The small sample segmentation method refers to using a small number of support images (Support images) with mask annotations and the query image (Query image) to be predicted as input, and outputting the semantic segmentation result on the query image.

[0003] In the related art, prototypes are usually used to represent typical information of foreground objects in the support image, and pixel-level features of the prototype and the query image are compared to predict the semantic segmentation result of the query image based on the feature comparison results. However, in the training process, the network in the related art usually treats the new category of objects as the background, resulting in the incorrect prediction of the new category of objects as the background during the network testing phase, thereby reducing the semantic segmentation accuracy of the network during the testing phase. Summary of the invention

[0004] In order to solve the above problems, the present application provides a semantic segmentation method, device, electronic device and storage medium.

[0005] On the one hand, the present application proposes a semantic segmentation method, the method comprising:

[0006] Obtain a support image and a query image to be segmented;

[0007] Inputting the support image and the query image into a semantic segmentation network to obtain a semantic segmentation result of the query image;

[0008] Among them, the semantic segmentation network is obtained by training the first branch network and the second branch network in the preset semantic segmentation network based on the sample support image annotated with the first sample category annotation result and the sample query image annotated with the second sample category annotation result. The first branch network is used to compare and learn the first sample image features of the sample support image and the second sample image features of the sample query image, and the second branch network is used to compare and learn at least two second sample sub-image features, and the at least two second sample sub-image features are obtained by clustering the sample background image features in the second sample image features.

[0009] On the other hand, an embodiment of the present application provides a semantic segmentation device, the device comprising:

[0010] An image acquisition module, used to acquire a support image and a query image to be segmented;

[0011] An input module, used for inputting the support image and the query image into a semantic segmentation network to obtain a semantic segmentation result of the query image;

[0012] Among them, the semantic segmentation network is obtained by training the first branch network and the second branch network in the preset semantic segmentation network based on the sample support image annotated with the first sample category annotation result and the sample query image annotated with the second sample category annotation result. The first branch network is used to compare and learn the first sample image features of the sample support image and the second sample image features of the sample query image, and the second branch network is used to compare and learn at least two second sample sub-image features, and the at least two second sample sub-image features are obtained by clustering the sample background image features in the second sample image features.

[0013] On the other hand, the present application proposes an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the semantic segmentation method as described above.

[0014] On the other hand, the present application proposes a computer-readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the semantic segmentation method as described above.

[0015] On the other hand, the present application proposes a computer program product, including a computer program, which implements the above-mentioned semantic segmentation method when executed by a processor.

[0016] The semantic segmentation method, apparatus, device and storage medium proposed in the embodiment of the present application, through the sample support image annotated with the first sample category annotation result and the sample query image annotated with the second sample category annotation result, the first branch network and the second branch network in the preset semantic segmentation network are trained to obtain the semantic segmentation network, wherein the first branch network is used to compare and learn the first sample image features of the sample support image and the second sample image features of the sample query image, and the second branch network is used to compare and learn at least two second sample sub-image features, and the at least two second sample sub-image features are obtained by clustering the sample background image features in the second sample image features. Since the training process of the semantic segmentation network can not only learn the comparison between the first sample image features of the sample support image with known categories and the second sample image features of the sample query image, but also learn the comparison between at least two second sample sub-image features with unknown categories, it enriches the feature comparison learning method of small sample semantic segmentation, can help the network learn more accurate feature representation, fully explore potential categories, avoid treating new category targets as background during training, thereby alleviating system bias problems and improving the training accuracy of the semantic segmentation network. In addition, using the semantic segmentation network with higher training accuracy can improve the semantic segmentation accuracy of query images in small sample semantic segmentation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a schematic diagram of an implementation environment of a semantic segmentation method according to an exemplary embodiment.

[0019] Figure 2 It is a flowchart of a semantic segmentation method according to an exemplary embodiment.

[0020] Figure 3 The present invention is a flow chart of a method for training a semantic segmentation network according to an exemplary embodiment.

[0021] Figure 4 The figure is another flowchart of obtaining a semantic segmentation network according to an exemplary embodiment.

[0022] Figure 5 The present invention is a flowchart showing a method of obtaining a first sample semantic segmentation result of a sample query image according to an exemplary embodiment.

[0023] Figure 6 The figure is a flow chart of a method for acquiring features of at least two second sample sub-images according to an exemplary embodiment.

[0024] Figure 7 The diagram is a schematic diagram showing a method of clustering sample background image features to obtain at least two second sample sub-image features according to an exemplary embodiment.

[0025] Figure 8 The figure is a flow chart showing a method of obtaining a second sample fusion feature according to an exemplary embodiment.

[0026] Fig. 9 It is a flowchart of obtaining the above-mentioned semantic segmentation network according to an exemplary embodiment.

[0027] Fig.10 The present invention is a flowchart showing a method for obtaining a semantic segmentation result of the query image according to an exemplary embodiment.

[0028] Fig.11 The figure is a comparison diagram of semantic segmentation results according to an exemplary embodiment.

[0029] Fig.12 The figure is a block diagram of a semantic segmentation device according to an exemplary embodiment.

[0030] Fig.13 The figure is a hardware structure block diagram of a server of a semantic segmentation method according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0032] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0033] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0034] Specifically, the process of obtaining the semantic segmentation network in the embodiments of the present application involves artificial neural network technology in deep learning in AI.

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Figure 1 FIG. 1 is a schematic diagram showing an implementation environment of a semantic segmentation method according to an exemplary embodiment. Figure 1 As shown, the implementation environment may include at least a terminal 01 and a server 02. The terminal 01 and the server 02 may be directly or indirectly connected via wired or wireless communication, which is not limited in the present embodiment.

[0038] Specifically, the terminal 01 can be used to collect supporting images and query images to be segmented. Optionally, the terminal 01 can include but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0039] Specifically, the server 02 can be used to obtain a support image and a query image to be segmented; and to input the support image and the query image into a semantic segmentation network to obtain a semantic segmentation result of the query image. And to train a first branch network and a second branch network in a preset semantic segmentation network based on a sample support image annotated with a first sample category annotation result and a sample query image annotated with a second sample category annotation result to obtain a semantic segmentation network.

[0040] Optionally, the server 02 may be an independent physical device or a physical server. It may also be a server cluster or a distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0041] It should be noted that Figure 1 This is just an example. In other scenarios, other implementation environments may also be included. For example, a scenario including a terminal, the terminal obtains a supporting image and a query image to be segmented; and the terminal inputs the first image feature and the second image feature into a semantic segmentation network to obtain a semantic segmentation result of the query image. And the terminal trains the first branch network and the second branch network in the preset semantic segmentation network based on the sample supporting image annotated with the first sample category annotation result and the sample query image annotated with the second sample category annotation result to obtain a semantic segmentation network.

[0042] Figure 2 is a flowchart of a semantic segmentation method according to an exemplary embodiment. The method can be used for Figure 1 In the implementation environment of the embodiment. This specification provides the method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on routine or non-creative work. The order of steps listed in the embodiment is only one way of executing the steps among many steps, and does not represent the only execution order. When the actual system or server product is executed, it can be executed in the order of the method shown in the embodiment or the figure or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the method may include:

[0043] S101. Obtain a supporting image and a query image to be segmented.

[0044] Optionally, the supporting image is annotated with category annotation results of the target. For example, if the supporting image contains targets such as mobile phones, sheep, and people, corresponding category annotation results can be annotated for mobile phones, sheep, people, etc. Exemplarily, the category annotation results can be reflected by a foreground mask, which is used to describe the area occupied by the target in the supporting image, that is, the target in the supporting image is annotated to mark the area where the target is located in the supporting image. It should be noted that the area in the supporting image other than the area where the target is located (i.e., the unannotated area) can be used as a background mask.

[0045] Exemplarily, both the foreground mask and the background mask can be binary images with the same height and width as the supporting image. The pixels in the binary image correspond one-to-one to the pixels at the same position in the supporting image. When the pixel value of any pixel in the foreground mask is 1, it means that the pixel corresponding to the pixel in the supporting image belongs to the foreground, in other words, to the target; when the pixel value of any pixel in the foreground mask is 0, it means that the pixel corresponding to the pixel in the supporting image belongs to the background. When the pixel value of any pixel in the background mask is 1, it means that the pixel corresponding to the pixel in the supporting image belongs to the background; when the pixel value of any pixel in the background mask is 0, it means that the pixel corresponding to the pixel in the supporting image belongs to the foreground.

[0046] Optionally, the query image is an image to be subjected to semantic segmentation.

[0047] In one method, the support image and the query image may contain the same object. That is, the support image and the query image both contain objects corresponding to the object category to be identified. By using the information of the support image, the object in the query image can be segmented. For example, if the object contained in both the support image and the query image is "mobile phone", then when performing semantic segmentation on the query image, "mobile phone" can be segmented.

[0048] In another way, the support image and the query image may not contain the same target. That is, the target corresponding to the target category to be identified may not exist in both the support image and the query image. By using the information of the support image, the corresponding background in the query image can be segmented out. For example, if the target contained in the support image is a "mobile phone", and the query image does not contain the "mobile phone", then when performing semantic segmentation on the query image, the corresponding background can be segmented out.

[0049] S103. Input the above-mentioned supporting image and the above-mentioned query image into a semantic segmentation network to obtain a semantic segmentation result of the above-mentioned query image; wherein the above-mentioned semantic segmentation network is obtained by training a first branch network and a second branch network in a preset semantic segmentation network based on a sample supporting image annotated with a first sample category annotation result and a sample query image annotated with a second sample category annotation result, the above-mentioned first branch network is used for comparative learning of the first sample image features of the above-mentioned sample supporting image and the second sample image features of the above-mentioned sample query image, the above-mentioned second branch network is used for comparative learning of at least two second sample sub-image features, and the above-mentioned at least two second sample sub-image features are obtained by clustering the sample background image features in the above-mentioned second sample image features.

[0050] In an embodiment of the present application, after the first image feature and the second image feature are input into the semantic segmentation network, the semantic segmentation network processes the first image feature and the second image feature to obtain a semantic segmentation result of the query image.

[0051] In one embodiment, the semantic segmentation result may be a result that the target in the query image is a certain category. For example, the semantic segmentation result is that the target in the query image is a “sheep”.

[0052] In another manner, the semantic segmentation result may be the probability that the target in the query image belongs to a certain category. For example, the semantic segmentation result may be that the probability that the target in the query image is a "sheep" is 90%.

[0053] In another manner, the semantic segmentation result may be a mask image of the target in the query image, and the mask image is used to describe the location area where the target in the query image is located.

[0054] It should be noted that the application scenario of the embodiment of the present application is a small sample segmentation scenario, in which the number of supporting images is a small number of images. That is, through a pre-trained semantic segmentation network, with the help of information from a small number of annotated supporting images, the query image is semantically segmented to obtain the semantic segmentation result of the query image.

[0055] In an embodiment of the present application, since the training process of the semantic segmentation network can not only learn the comparison between the first sample image features of the sample support image whose category is known and the second sample image features of the sample query image, but also learn the comparison between at least two second sample sub-image features whose category is unknown, it enriches the feature comparison learning method of small sample semantic segmentation, can help the network learn more accurate feature representation, fully explore potential categories, and avoid treating targets of new categories as background during training, thereby alleviating system bias problems and improving the training accuracy of the semantic segmentation network. Using the semantic segmentation network with higher training accuracy, it is possible to avoid treating targets of new categories as background during testing, and can improve the determination accuracy of the semantic segmentation results of query images in small sample semantic segmentation scenarios.

[0056] The following introduces the training process of the semantic segmentation network.

[0057] Figure 3 FIG. 1 is a flow chart of a method for training a semantic segmentation network according to an exemplary embodiment. Figure 3 As shown, in an optional embodiment, the training method of the semantic segmentation network may include:

[0058] S201. Obtain the sample support image and the sample query image.

[0059] Optionally, the sample support image is annotated with a first sample category annotation result of a sample target. For example, if the support image contains targets such as a mobile phone, a sheep, and a person, corresponding category annotation results can be annotated for the mobile phone, the sheep, the person, etc. The sample query image is an image to be subjected to semantic segmentation, which is annotated with a second sample category annotation result of a sample target.

[0060] Optionally, the sample support image and the sample query image contain the same sample target. That is, the sample support image and the sample query image both contain sample targets corresponding to the sample target category to be identified. By using the information of the sample support image, the sample target in the sample query image can be segmented out. For example, if the sample target contained in both the sample support image and the sample query image is "TV", then when semantic segmentation is performed on the sample query image, "TV" can be segmented out.

[0061] Optionally, both the first sample category labeling result and the second sample category labeling result can be reflected by a sample foreground mask. Taking the sample support image as an example, the sample foreground mask is used to describe the sample area occupied by the sample target in the sample support image (the sample foreground area of ​​the sample support image), that is, the sample target (i.e., the foreground) in the sample support image is labeled to mark the sample area where the sample target in the sample support image is located. It should be noted that the area other than the area where the sample target is located in the sample support image (i.e., the unlabeled area) can be used as the sample background mask of the sample support image. Taking the sample query image as an example, the sample foreground mask is used to describe the sample area occupied by the sample target in the sample query image (the sample foreground area of ​​the sample query image), that is, the sample target (i.e., the foreground) in the sample query image is labeled to mark the sample area where the sample target in the sample query image is located. It should be noted that the area other than the area where the sample target is located in the sample query image (i.e., the unlabeled area) can be used as the sample background mask of the sample query image.

[0062] Exemplarily, the sample foreground mask and the sample background mask may both be binary images having the same height and width as the sample support image or the sample query image.

[0063] It should be noted that the application scenario of the embodiment of the present application is a small sample segmentation scenario. In the small sample segmentation scenario, the number of sample support images is only a few, that is, the preset neural network is trained by a small number of labeled sample support images and sample query images to obtain the semantic segmentation network.

[0064] S203. Based on the first branch network, comparative learning is performed on the first sample image features and the second sample image features to obtain a first sample semantic segmentation result of the sample query image.

[0065] Figure 4 FIG. 1 is another flowchart of obtaining a semantic segmentation network according to an exemplary embodiment. Figure 4 As shown, in one embodiment, before the sample support image and the sample query image are input into the preset semantic segmentation network, the first sample image feature of the sample support image and the second sample image feature of the query image can be extracted through a pre-trained feature extraction network. Exemplarily, the feature extraction network can be trained by deep neural networks such as Residual Neural Network (ResNet), Visual Geometry Group Network (VGG), and Wide Residual Network (Wide ResNet).

[0066] Exemplarily, both the first sample image feature and the second sample image feature may be feature maps.

[0067] In another embodiment, a feature extraction module may be provided in the preset semantic segmentation network, and the first sample image feature of the sample support image and the second sample image feature of the query image may be extracted by the feature extraction module.

[0068] Optionally, continue as Figure 4 As shown, the preset neural network may include two branch networks, namely, a first branch network and a second branch network. The first branch network is used to perform comparative learning on the first sample image feature and the second sample image feature. Since the sample support image and the sample query image are both labeled with categories, the categories are all known, that is, the first branch network can be understood as a category-known branch network, which can learn the comparison between the first sample image feature and the second sample image feature whose categories are known. The second branch network is used to perform comparative learning on at least two second sample sub-image features. Since the at least two second sample sub-image features are obtained by clustering the sample background image features, the annotation of the sample query image is to mark out the target sample, that is, to mark the foreground, but not the background area, and the categories of at least two second sample sub-image features are unknown, that is, the second branch network can be understood as a category-agnostic branch network, which is used to learn the comparison between at least two second sample sub-image features corresponding to the sample background image features of the sample query image. The above new learning paradigm enriches the feature comparison learning method, which can help the network learn more accurate feature representation, fully explore potential categories, and avoid treating new categories of targets as background during training, thereby alleviating system bias problems and improving the training accuracy of semantic segmentation networks.

[0069] Figure 5 is a flowchart showing a method of obtaining a first sample semantic segmentation result of a sample query image according to an exemplary embodiment. Figure 5 As shown, in an optional embodiment, in the above step S203, obtaining the first sample semantic segmentation result of the above sample query image may include:

[0070] S2031. Input the first sample image features and the first sample category labeling results into the first branch network.

[0071] S2033. Based on the first branch network, global average pooling is performed on the first sample image features and the first sample category annotation results to obtain a sample foreground prototype corresponding to the sample foreground area of ​​the sample support image; the sample foreground area of ​​the sample support image is the area where the first sample category annotation result is located.

[0072] S2035. Based on the first branch network, the sample foreground prototype and the second sample image feature are fused to obtain a first sample fusion feature.

[0073] S2037. Encode the first sample fusion features based on the first branch network to obtain the first sample semantic segmentation result.

[0074] Optionally, in the above step S2031, continue as follows Figure 4 As shown, the first sample image feature and the first sample category labeling result can be used as input of the first branch network.

[0075] Optionally, in the above step S2033, continue as follows Figure 4 As shown, the first branch network can perform global average pooling processing on the first sample image features and the first sample category annotation results to obtain the sample foreground prototype. The prototype can refer to a relationship mapping between the input image and the target category, and can also be understood as an abstract expression of the target category in the image. The sample foreground prototype can refer to a relationship mapping between the sample support image and the sample category, and can also be understood as an abstract expression of the sample category in the sample support image.

[0076] For example, taking the first sample category annotation result used to describe the sample area occupied by the sample target in the supporting image in the sample supporting image, that is, the sample foreground mask as an example, the following formula can be used to calculate the sample foreground prototype:

[0077]

[0078] in, refers to the sample foreground prototype, refers to the first sample image feature, refers to the first sample category labeling result (for example, sample foreground mask), and ⊙ refers to the element-level multiplication operation.

[0079] Optionally, in the above step S2035, the first branch network can Assigning each spatial position of the second sample image feature to the sample query image, and learning feature comparison to identify the foreground object presented in the query image. In one embodiment, in the above step S2035, the above obtaining the first sample fusion feature may include:

[0080] Based on the first branch network, the size of the sample foreground prototype is expanded to the size of the second sample image feature to obtain an expanded sample foreground prototype.

[0081] Based on the first branch network, the expanded sample foreground prototype and the second sample image feature are fused to obtain the first sample fusion feature.

[0082] For example, continue as Figure 4 As shown, the sample foreground prototype The size of the second sample image feature (F q ) of the same size, making them the same shape, and then q In the channel dimension, the expanded sample foreground prototype and the second sample image features are fused and spliced ​​together to obtain the first sample fusion feature. The fusion formula can be as follows:

[0083]

[0084] Among them, E s Refers to the sample foreground prototype The expansion operation, F q refers to the second sample image feature, Refers to fusion, X sq Refers to the first sample fusion feature.

[0085] In the embodiment of the present application, by expanding the size of the sample foreground prototype to the size of the second sample image feature and fusing the expanded sample foreground prototype and the second sample image feature, the determination accuracy of the first sample fusion feature can be improved, thereby improving the determination accuracy of the first sample semantic segmentation result in the small sample semantic segmentation scenario.

[0086] In a feasible embodiment, in order to obtain a better semantic segmentation result on the sample query image, in the above step S2037, continue as follows Figure 4 As shown, the first sample fusion feature can be encoded through the convolution module in the first branch network to obtain the first sample semantic segmentation result. The formula can be as follows:

[0087]

[0088] Among them, X sq Refers to the first sample fusion feature, g φ refers to convolution, Refers to the first sample semantic segmentation result.

[0089] Optionally, g φ The pixel-level feature comparison can be performed by binary classification. During the convolution coding process, it is verified whether the pixels in the second sample image feature match the prototype at the corresponding position of the expanded sample foreground prototype. If they match, then is 1, otherwise, is 0.

[0090] In one embodiment, the first sample semantic segmentation result may be a result that the sample target in the sample query image is a certain category. For example, the first sample semantic segmentation result is that the sample target in the sample query image is “sheep”.

[0091] In another embodiment, the first sample semantic segmentation result may be the probability that the sample target in the sample query image belongs to a certain category. For example, the first sample semantic segmentation result may be that the probability that the sample target in the sample query image is "sheep" is 90%.

[0092] In another manner, the first sample semantic segmentation result may be a sample mask image of the sample target in the sample query image, and the sample mask image is used to describe the location area where the sample target in the sample query image is located.

[0093] In an embodiment of the present application, a first branch network performs global average pooling processing on the first sample image features and the first sample category annotation results to obtain a sample foreground prototype, and the sample foreground prototype and the second sample image features are fused to obtain a first sample fusion feature. The first sample fusion feature is encoded to obtain the first sample semantic segmentation result, thereby realizing comparative learning of the first sample image features and the second sample image features whose categories are known, enriching the feature comparison learning method in small sample semantic segmentation scenarios, helping the network to learn more accurate feature representations, fully tapping potential categories, and avoiding treating new categories of targets as background during training, thereby alleviating system bias problems and improving the training accuracy of the semantic segmentation network.

[0094] S205. Based on the second branch network, comparative learning is performed on the features of the at least two second sample sub-images to obtain a second sample semantic segmentation result of the sample query image.

[0095] Figure 6 FIG. 1 is a flow chart showing a method for acquiring at least two second sample sub-image features according to an exemplary embodiment. Figure 6 As shown, in an optional embodiment, the method may further include obtaining the at least two second sample sub-image features. The method for obtaining the at least two second sample sub-image features may include:

[0096] S301. The area of ​​the sample query image except the area where the second sample category annotation result is located is used as the sample background area of ​​the sample query image.

[0097] S303. Obtain the sample background image features corresponding to the sample background area in the second sample image features.

[0098] S305. Clustering the sample background image features based on the distances between the features in the sample background image features to obtain the at least two second sample sub-image features.

[0099] Optionally, in order to obtain a sample background prototype of a sample query image, the sample background region of the sample query image may be clustered, thereby dividing the sample background image features corresponding to the sample background region into at least two second sample sub-image features.

[0100] Optionally, when the second sample category annotation result is used to describe the sample area occupied by the sample target in the sample query image, in the above step S301, the sample background area in the sample query image can be determined according to the second sample category annotation result. Exemplarily, the area in the sample query image other than the area where the second sample category annotation result is located can be used as the sample background area, that is, the area other than the area where the sample foreground marked in the sample query image is located can be used as the sample background area.

[0101] Optionally, in the above step S303, the image features corresponding to the sample background region may be extracted from the second sample image features to obtain the sample background image features, wherein the sample background image features may be used to describe the depth features corresponding to the sample background region.

[0102] In one way, in the above step S305, the k-means clustering algorithm (k-means clustering algorithm) can be used to divide the sample background image features into several blocks to obtain at least two second sample sub-image features. As an example, the k-means clustering algorithm can be used to cluster the distances between features in the sample background image features in the feature space to obtain at least two feature clusters, and the at least two feature clusters are used as the at least two second sample sub-image features. The features in the at least two feature clusters are connected as closely as possible, and the distances between different feature clusters are as large as possible. In another way, in the above step S305, the sample background image features can also be divided into several blocks by using a fixed grid division method.

[0103] Exemplarily, the sample background image feature may be a sample background feature map, and the second sample sub-image feature may be a second sample sub-feature map. In this case, clustering the sample background image features is equivalent to dividing the sample background image features into a plurality of image feature regions.

[0104] Figure 7 FIG. 1 is a schematic diagram showing a method of clustering sample background image features to obtain at least two second sample sub-image features according to an exemplary embodiment. Figure 7As shown in Figure 2, the division areas generated by clustering using the k-means clustering algorithm are more reasonable, and each area has relatively clear semantic information. Figure 7 Take the first column in as an example to illustrate. Figure 7 The sample foreground in the first column is "bicycle", and the sample background area is the area other than the area where the "bicycle" is located. The k-means clustering algorithm is used to cluster the distances between the features in the sample background image features in the feature space to obtain three blocks (block 1, block 2, block 3), which indicate three second sample sub-image features.

[0105] In an embodiment of the present application, since the sample background area is an area in the sample query image other than the sample area occupied by the sample target described by the second sample category annotation result (the area other than the sample foreground area), the sample background area is unlabeled, and the k-means clustering algorithm is used to divide the sample background area in an unsupervised manner, that is, the sample background image features are divided into several blocks to obtain multiple category-unknown proxy features (i.e., at least two second sample sub-image features), so that the corresponding sample background prototype can be obtained for each block later. Since each pixel in the block should be similar to the corresponding background prototype, it is convenient to find prototypes for unlabeled targets, and it is also convenient to perform feature comparison in a self-comparison manner for multiple category-unknown proxy features according to the second branch network later, so as to fully explore potential categories and avoid treating new categories of targets as backgrounds during training, thereby alleviating the system bias problem in small sample semantic segmentation scenarios, thereby improving the training accuracy of the semantic segmentation network.

[0106] Figure 8 is a flow chart showing a method of obtaining a second sample fusion feature according to an exemplary embodiment. Figure 8 As shown, in an optional embodiment, in the above step S205, obtaining the second sample fusion feature may include:

[0107] S2051. Input the at least two second sample sub-image features into the second branch network.

[0108] S2053. Based on the second branch network, perform global average pooling processing on the at least two second sample sub-image features and the second sample category labeling results to obtain sample background prototypes corresponding to the at least two second sample sub-image features.

[0109] Optionally, in the above step S2051, at least two second sample sub-image features may be used as inputs of the second branch network, and the second branch network performs self-alignment between category-agnostic proxy features on the at least two second sample sub-image features.

[0110] Optionally, in the above step S2053, continue as follows Figure 4 As shown, the second branch network can perform global average pooling processing on the second sample image features and the second sample category annotation results to obtain sample background prototypes corresponding to at least two second sample sub-image features. For example, if there are three at least two second sample sub-image features, three sample background prototypes will be generated.

[0111] Exemplarily, the sample background prototype may refer to a relationship mapping between the sample query image and the sample category, and may also be understood as an abstract expression of the sample category in the sample query image.

[0112] For example, taking the second sample category annotation result used to describe the sample area occupied by the sample target in the sample query image, that is, the foreground mask, as an example, the sample background prototype can be calculated using the following formula:

[0113]

[0114] in, Refers to the sample background prototype, refers to at least two second sample image features, refers to the second sample category labeling result, ⊙ refers to the element-level multiplication operation, and h is The height of w is width.

[0115] S2055. Based on the second branch network, the at least two second sample sub-image features and the sample background prototypes corresponding to the at least two second sample sub-image features are fused to obtain a second sample fusion feature.

[0116] In one embodiment, the second branch network may include a fusion unit, which fuses at least two second sample sub-image features with sample background prototypes corresponding to the at least two second sample sub-image features to obtain the second sample fusion feature.

[0117] In another embodiment, in the above step S2055, obtaining the second sample fusion feature may include:

[0118] Based on the second branch network, the size of each sample background prototype is expanded to the size of the corresponding second sample sub-image feature, so as to obtain the expanded sample background prototype corresponding to each sample background prototype.

[0119] The expanded sample background prototype corresponding to each of the sample background prototypes and the corresponding second sample sub-image features are fused to obtain the second sample fusion features.

[0120] Optionally, each sample background prototype can be extended Size, filling The corresponding position of 1 is obtained by expanding each background prototype to obtain a regional sample background prototype, that is, the expanded sample background prototype. Then, all the expanded sample background prototypes and its corresponding second sample sub-image feature are connected in series in the depth dimension, where "corresponding second sample sub-image feature" refers to the second sample sub-image feature used to generate the sample background prototype. In this way, the positive construction of each sample background prototype is achieved (i.e., the sample background prototype and the corresponding second sample sub-image feature). At the same time, a sample background prototype can be randomly selected from all the sample background prototypes and densely paired with each position of the foreground feature in the sample query image (i.e., the feature corresponding to the sample foreground area). This pairing method will obtain a trivial solution, that is, a mask with 1 for the entire image.

[0121] Exemplarily, the following formula may be used to fuse the extended sample background prototype and the corresponding second sample sub-image features corresponding to each sample background prototype:

[0122]

[0123] Among them, X qq refers to the second sample fusion feature, Refers to the sample background prototypes corresponding to at least two second sample sub-image features, F q refers to the second sample sub-image feature, E q Refers to the expansion operation of the sample background prototype.

[0124] In the embodiment of the present application, by expanding the size of the sample background prototype to the size of the corresponding second sample image feature and fusing the expanded sample background prototype and the corresponding second sample image feature, the determination accuracy of the second sample fusion feature can be improved, thereby improving the determination accuracy of the second sample semantic segmentation result.

[0125] S2057. Encode the second sample fusion features based on the second branch network to obtain the second sample semantic segmentation result.

[0126] In a feasible embodiment, in order to obtain a better semantic segmentation result on the sample image, in the above step S2057, continue as follows Figure 4 As shown, a convolution module can be set in the second branch network, and the second sample fusion feature is encoded by the convolution module to obtain the second sample semantic segmentation result. The formula can be as follows:

[0127]

[0128] Among them, X qq refers to the second sample fusion feature, g φ refers to convolution, Refers to the second sample semantic segmentation result.

[0129] Optionally, g φ The pixel-level feature comparison is performed by binary classification. During the convolution coding process, it is verified whether the pixels in the second sample sub-image feature match the prototype at the corresponding position of the expanded sample background prototype. If they match, then is 1, otherwise, is 0.

[0130] In one embodiment, the second sample semantic segmentation result may be a result that the sample target in the sample query image is a certain category. For example, the second sample semantic segmentation result is that the sample target in the sample query image is “sheep”.

[0131] In another manner, the second sample semantic segmentation result may be the probability that the sample target in the sample query image belongs to a certain category. For example, the second sample semantic segmentation result may be that the probability that the sample target in the sample query image is "sheep" is 90%.

[0132] In another manner, the second sample semantic segmentation result may be a sample mask image of the sample target in the sample query image, and the sample mask image is used to describe the location area where the sample target in the sample query image is located.

[0133] In an embodiment of the present application, a global average pooling process is performed on at least two second sample sub-image features and the above-mentioned second sample category annotation results through a second branch network to obtain sample background prototypes corresponding to at least two second sample sub-image features, fuse at least two second sample sub-image features and their corresponding sample background prototypes to obtain second sample fusion features, encode the second sample fusion features, and obtain a second sample semantic segmentation result, thereby realizing self-contrast learning between at least two second sample sub-image features that are category-agnostic, minimizing each proxy feature and the corresponding feature area, maximizing foreground features and any category-agnostic proxy features, enriching the feature comparison learning method in small sample semantic segmentation scenarios, helping the network learn more accurate feature representations, fully exploring potential categories, and avoiding treating new categories of targets as backgrounds during training, thereby alleviating the system bias problem in small sample semantic segmentation scenarios and improving the training accuracy of the semantic segmentation network; and the corresponding sample background prototype can be obtained for each block. Since each pixel in the block should be similar to the corresponding background prototype, the difficulty of finding unlabeled target prototypes is reduced, and the training accuracy of the semantic segmentation network is further improved.

[0134] S207. According to the semantic segmentation result of the first sample, the semantic segmentation result of the second sample and the category labeling result of the second sample, the network parameters in the preset semantic segmentation network are updated until the network parameters meet the preset conditions to obtain the semantic segmentation network.

[0135] Fig. 9 is a flowchart of obtaining the above-mentioned semantic segmentation network according to an exemplary embodiment. Fig. 9 As shown, in an optional embodiment, in the above step S207, obtaining the above semantic segmentation network may include:

[0136] S2071. Determine a first loss value based on the semantic segmentation result of the first sample and the category labeling result of the second sample.

[0137] S2073. Determine a second loss value based on the semantic segmentation result and the non-category labeling result of the second sample; the non-category labeling result represents the difference between the category labeling result of the second sample and 1.

[0138] S2075. Determine a target loss value based on the first loss value and the above-mentioned second loss value.

[0139] S2077. Based on the above target loss value, the network parameters in the above preset semantic segmentation network are updated until the above network parameters meet the preset conditions to obtain the above semantic segmentation network.

[0140] In the present application embodiment, continue as Figure 4 As shown, Based on X sq The encoded output of , which combines the first sample image features of the sample foreground prototype and the sample query, and By X qq It is obtained by using at least two second sample sub-image features of the sample background prototype and the sample background image features. and The common feature of the foreground and background is the feature comparison, whether it is X sq Or X qq , if the comparison is on the same semantics (i.e. the same foreground class or background cluster), the network should output a prediction of 1 at these locations, otherwise it should output a prediction of 0. That is, for the category-aware branch network, its target label should be the mask M of the sample query image q (i.e., the second sample category labeling result), and for the category-agnostic network branch, its target label should be M q .

[0141] Therefore, the semantic segmentation network can be obtained by training using cross entropy loss. Optionally, in the step S2071, the first loss value can be determined according to the similarity between the semantic segmentation result of the first sample and the category labeling result of the second sample.

[0142] Optionally, in the above step S2073, the semantic segmentation result of the second sample and the non-category labeling result (i.e. ) to determine the second loss value.

[0143] Optionally, in the above step S2075, it is possible to determine the first weight of the first loss value and the second weight of the above second loss value, calculate the first product between the first loss value and the first weight, calculate the second product between the second loss value and the second weight, and take the sum of the first product and the second product as the target loss value.

[0144] Optionally, in the above step S2077, the network parameters in the preset semantic segmentation network can be updated based on the above target loss value until the above network parameters meet the preset conditions to obtain the semantic segmentation network. The calculation formula of the target loss value can be as follows:

[0145]

[0146] Among them, L is the target loss value, (1-λ) is the first weight, λ is the second weight, L1 is the first loss value, and L2 is the second loss value.

[0147] In the present application embodiment, due to Based on Xsq The encoded output of , which combines the first sample image features of the sample foreground prototype and the sample query, and By X qq The network parameters in the preset semantic segmentation network are updated by using the first loss value determined by the first sample semantic segmentation result and the above-mentioned second sample category labeling result, and the second loss value determined by the second sample semantic segmentation result and the non-category labeling result, obtained by using the sample background prototype and at least two second sample sub-image features of the sample background image features. The training process of the semantic segmentation network can learn the comparison between the first sample image features and the second sample image features whose categories are known, and can also learn the comparison between at least two second sample sub-image features whose categories are unknown. This new learning paradigm enriches the feature comparison learning method, can help the network learn more accurate feature representation, fully explore potential categories, and avoid treating new categories of targets as backgrounds during training, thereby alleviating the system bias problem in small sample semantic segmentation scenarios and improving the training accuracy of the semantic segmentation network.

[0148] Fig.10 is a flowchart showing a method of obtaining a semantic segmentation result of the query image according to an exemplary embodiment. Fig.10 In an optional embodiment, in the above step S103, the above supporting image is annotated with a category annotation result, and the above supporting image and the above query image are input into a semantic segmentation network to obtain a semantic segmentation result of the above query image, which may include:

[0149] S1031. Based on the semantic segmentation network, a global average pooling process is performed on the first image feature of the supporting image and the category labeling result to obtain a foreground prototype corresponding to the foreground area of ​​the supporting image; the foreground area is the area where the category labeling result is located.

[0150] S1033. Based on the semantic segmentation network, the foreground prototype and the second image features of the query image are fused to obtain fused features.

[0151] S1035. Encode the above fusion features based on the above semantic segmentation network to obtain the above semantic segmentation results.

[0152] In one embodiment, the query image can be input into a pre-trained feature extraction network to extract the first image feature of the support image, and the support image can be input into a pre-trained feature extraction network to extract the second image feature of the query image. Exemplarily, the feature extraction network can be trained by a deep neural network such as a Residual Neural Network (ResNet), a Visual Geometry Group Network (VGG), or a Wide Residual Network (Wide ResNet).

[0153] Exemplarily, both the first image feature and the second image feature may be feature maps.

[0154] In another embodiment, a feature extraction module may also be provided in the semantic segmentation network, and the first image feature of the supporting image and the second image feature of the query image may be extracted by the feature extraction module.

[0155] It should be noted that, although in the training stage of the semantic segmentation network, the first branch network is used to compare and learn the first sample image features of the sample support image whose category is known and the second sample image features of the sample query image, and the second branch network is used to compare and learn the at least two second sample sub-image features whose category is unknown, in the actual use stage of the semantic segmentation network, since the query image is the image to be segmented, the query image is not labeled. Therefore, in the actual use stage of the semantic segmentation network, there is no need for a category-agnostic branch network, and only the category-agnostic branch network needs to be used to process the first image features and the above-mentioned second image features to obtain the semantic segmentation result of the query image.

[0156] It should be noted that, in the actual use stage of the semantic segmentation network, the principle and training process type of the category-aware branch network. Optionally, in the above step S1031, the category-aware branch network in the semantic segmentation network can be used to perform global average pooling processing on the above first image features and the above category annotation results to obtain a foreground prototype corresponding to the foreground area of ​​the support image, where the foreground area is the area where the category annotation results are located. The method of calculating the foreground prototype is similar to the method of calculating the sample foreground prototype in the above step S2033, and will not be described in detail here.

[0157] Optionally, in the above step S1033, the foreground prototype and the second image feature may be fused based on the category-aware branch network in the above semantic segmentation network to obtain a fused feature. The fusion process is similar to the above step S2035 and will not be described in detail here.

[0158] Optionally, in the above step S1035, the above fusion features may be encoded based on the encoding network in the semantic segmentation network to obtain the above semantic segmentation result. The encoding process is similar to the above step S2037, and will not be repeated here.

[0159] In the embodiment of the present application, since the semantic segmentation network can learn both the comparison between category-agnostic features and the comparison between category-agnostic features during training, the semantic segmentation network can have the opportunity to predict unknown new class targets appearing in the background as foregrounds instead of always predicting them as backgrounds, thereby improving the semantic segmentation accuracy in small sample segmentation scenarios; in addition, in the actual use stage of the semantic segmentation network, there is no need for a category-agnostic branch network, and only the category-agnostic branch network needs to be used to process the first image features and the above-mentioned second image features to obtain the semantic segmentation results of the query image. Therefore, it does not bring any additional computational costs and hyperparameters, that is, it does not consume additional system resources and does not increase additional reasoning time.

[0160] Fig.11 is a comparison diagram of semantic segmentation results according to an exemplary embodiment. Existing methods may segment a small part of new category objects or fail to segment them at all due to prior bias, for example Fig.11 The mobile phone in the second column, the bus in the fourth column, and the person in the seventh column. Even if the background of the query image contains some other objects, the method in the embodiment of the present application can make a correct prediction, for example Fig.11 When the scale difference between the objects in the support image and the query image is large, the method selected in the embodiment of the present application can also well segment the foreground object in the query image, for example Fig.11 The sheep in the first column of the figure, the potted plants in the fifth column. It should be noted that when some small objects appear in the foreground (such as the person on the TV in the sixth column), the method selected in the embodiment of the present application may predict the person as the background, because the person on the TV is also a type of target.

[0161] In a feasible embodiment, at least one of the first sample semantic segmentation result in S203, the second sample semantic segmentation result in S205, the first sample fusion feature in S2035, and the second sample fusion feature in S2055 can be stored in the blockchain system.

[0162] Fig.12 is a block diagram of a semantic segmentation device according to an exemplary embodiment. Fig.12 As shown, the device may include at least:

[0163] The image acquisition module 401 is used to acquire a supporting image and a query image to be segmented.

[0164] The input module 403 is used to input the support image and the query image into a semantic segmentation network to obtain a semantic segmentation result of the query image.

[0165] Among them, the above-mentioned semantic segmentation network is obtained by training the first branch network and the second branch network in the preset semantic segmentation network based on the sample support image annotated with the first sample category annotation results and the sample query image annotated with the second sample category annotation results. The above-mentioned first branch network is used to compare and learn the first sample image features of the above-mentioned sample support image and the second sample image features of the above-mentioned sample query image, and the above-mentioned second branch network is used to compare and learn at least two second sample sub-image features, and the above-mentioned at least two second sample sub-image features are obtained by clustering the sample background image features in the above-mentioned second sample image features.

[0166] In an optional embodiment, the supporting image is annotated with a category annotation result, and the input module 403 may include:

[0167] A foreground prototype acquisition unit is used to perform global average pooling processing on the first image feature of the supporting image and the category annotation result based on the semantic segmentation network to obtain a foreground prototype corresponding to the foreground area of ​​the supporting image; the foreground area is the area where the category annotation result is located.

[0168] The fusion feature acquisition unit is used to fuse the foreground prototype and the second image feature of the query image based on the semantic segmentation network to obtain a fusion feature.

[0169] The semantic segmentation result determination unit is used to encode the above fusion features based on the above semantic segmentation network to obtain the above semantic segmentation result.

[0170] In an optional embodiment, the above device further includes:

[0171] The sample image acquisition module is used to acquire the sample support image and the sample query image.

[0172] The first sample semantic segmentation result acquisition module is used to compare and learn the first sample image features and the second sample image features based on the first branch network to obtain the first sample semantic segmentation result of the sample query image.

[0173] The second sample semantic segmentation result acquisition module is used to perform comparative learning on the features of the at least two second sample sub-images based on the second branch network to obtain the second sample semantic segmentation result of the sample query image.

[0174] An updating module is used to update the network parameters in the preset semantic segmentation network according to the first sample semantic segmentation result, the second sample semantic segmentation result and the second sample category labeling result, until the network parameters meet the preset conditions to obtain the semantic segmentation network.

[0175] In an optional embodiment, the above device further includes:

[0176] The sample background region determination module is used to use the region in the sample query image, except the region where the second sample category annotation result is located, as the sample background region of the sample query image.

[0177] The background image feature acquisition module is used to acquire the sample background image features corresponding to the sample background area in the second sample image features.

[0178] The clustering module is used to cluster the sample background image features based on the distances between the features in the sample background image features to obtain the at least two second sample sub-image features.

[0179] In an optional embodiment, the first sample semantic segmentation result acquisition module may include:

[0180] The first input unit is used to input the first sample image feature and the first sample category labeling result into the first branch network.

[0181] A sample foreground prototype acquisition unit is used to perform global average pooling processing on the first sample image features and the first sample category annotation results based on the first branch network to obtain a sample foreground prototype corresponding to the sample foreground area of ​​the sample support image; the sample foreground area of ​​the sample support image is the area where the first sample category annotation results are located.

[0182] The first sample fusion feature acquisition unit is used to fuse the sample foreground prototype and the second sample image feature based on the first branch network to obtain a first sample fusion feature.

[0183] The first encoding unit is used to encode the first sample fusion feature based on the first branch network to obtain the first sample semantic segmentation result.

[0184] In an optional embodiment, the first sample fusion feature acquisition unit includes:

[0185] The first expansion subunit is used to expand the size of the sample foreground prototype to the size of the second sample image feature based on the first branch network to obtain the expanded sample foreground prototype.

[0186] The first fusion subunit is used to fuse the expanded sample foreground prototype and the second sample image feature based on the first branch network to obtain the first sample fusion feature.

[0187] In an optional embodiment, the second sample semantic segmentation result acquisition module includes:

[0188] The second input unit is used to input the at least two second sample sub-image features into the second branch network.

[0189] The sample background prototype acquisition unit is used to perform global average pooling processing on the at least two second sample sub-image features and the second sample category labeling results based on the second branch network to obtain the sample background prototypes corresponding to the at least two second sample sub-image features.

[0190] The second sample fusion feature acquisition unit is used to fuse the at least two second sample sub-image features and the sample background prototypes corresponding to the at least two second sample sub-image features based on the second branch network to obtain the second sample fusion feature.

[0191] The second encoding unit is used to encode the second sample fusion feature based on the second branch network to obtain the second sample semantic segmentation result.

[0192] In an optional embodiment, the second sample fusion feature acquisition unit includes:

[0193] The second expansion subunit is used to expand the size of each sample background prototype to the size of the corresponding second sample sub-image feature based on the second branch network to obtain the expanded sample background prototype corresponding to each sample background prototype.

[0194] The second fusion subunit is used to fuse the expanded sample background prototype corresponding to each of the sample background prototypes and the corresponding second sample sub-image features to obtain the second sample fusion features.

[0195] In an optional embodiment, the update module includes:

[0196] The first loss value determining unit is used to determine a first loss value based on the first sample semantic segmentation result and the second sample category labeling result.

[0197] A second loss value determining unit is used to determine a second loss value based on the second sample semantic segmentation result and the non-category labeling result; the non-category labeling result represents the difference between the second sample category labeling result and 1.

[0198] The target loss value determining unit is used to determine the target loss value according to the first loss value and the second loss value.

[0199] The semantic segmentation network determination unit is used to update the above network parameters based on the above target loss value until the above network parameters meet the preset conditions to obtain the above semantic segmentation network.

[0200] It should be noted that the device embodiment provided in the embodiments of the present application and the above-mentioned method embodiment are based on the same inventive concept.

[0201] An embodiment of the present application also provides an electronic device for semantic segmentation, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the semantic segmentation method provided in the above method embodiment.

[0202] An embodiment of the present application also provides a computer-readable storage medium, which can be set in a terminal to store at least one instruction or at least one program related to semantic segmentation in an implementation method embodiment, and the at least one instruction or at least one program is loaded and executed by a processor to implement the semantic segmentation method provided in the above-mentioned method embodiment.

[0203] Optionally, in the embodiment of this specification, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the above storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0204] The memory of the embodiment of this specification can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, applications required for functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0205] The embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the semantic segmentation method provided by the above method embodiment.

[0206] The semantic segmentation method embodiment provided in the embodiment of the present application can be executed in a terminal, a computer terminal, a server or a similar computing device. Taking running on a server as an example, Fig.13 FIG. 1 is a hardware structure block diagram of a server of a semantic segmentation method according to an exemplary embodiment. Fig.13 As shown, the server 500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 510 (the central processing unit 510 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 530 for storing data, and one or more storage media 520 (such as one or more mass storage devices) for storing application programs 523 or data 522. Among them, the memory 530 and the storage medium 520 can be short-term storage or permanent storage. The program stored in the storage medium 520 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 510 can be configured to communicate with the storage medium 520 and execute a series of instruction operations in the storage medium 520 on the server 500. The server 500 may also include one or more power supplies 560, one or more wired or wireless network interfaces 550, one or more input and output interfaces 540, and / or, one or more operating systems 521, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM etc.

[0207] The input / output interface 540 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 500. In one example, the input / output interface 540 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 540 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0208] It can be understood by those skilled in the art that Fig.13 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.13 More or fewer components as shown, or with Fig.13 Different configurations shown.

[0209] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0210] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0211] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0212] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A semantic segmentation method, characterized in that: The method comprises: Obtain a support image and a query image to be segmented; Inputting the supporting image and the query image into a semantic segmentation network to obtain a semantic segmentation result of the query image; the supporting image is annotated with a category annotation result, and inputting the supporting image and the query image into a semantic segmentation network to obtain a semantic segmentation result of the query image, including: performing global average pooling processing on the first image feature of the supporting image and the category annotation result based on the semantic segmentation network to obtain a foreground prototype corresponding to the foreground area of ​​the supporting image; the foreground area is the area where the category annotation result is located; fusing the foreground prototype and the second image feature of the query image based on the semantic segmentation network to obtain a fused feature; encoding the fused feature based on the semantic segmentation network to obtain the semantic segmentation result; Among them, the semantic segmentation network is obtained by updating the network parameters in the preset semantic segmentation network based on the target loss value, the target loss value is determined based on the first loss value and the second loss value, the first loss value is determined based on the first sample semantic segmentation result and the second sample category annotation result annotated in the sample query image, and the second loss value is determined based on the second sample semantic segmentation result and the non-category annotation result, and the non-category annotation result represents the difference between the second sample category annotation result and 1; the first sample semantic segmentation result is obtained by comparing and learning the first sample image features of the sample support image annotated with the first sample category annotation result and the second sample image features of the sample query image based on the first branch network in the preset semantic segmentation network; the second sample semantic segmentation result is obtained by comparing and learning at least two second sample sub-image features based on the second branch network in the preset semantic segmentation network, and the at least two second sample sub-image features are obtained by clustering the sample background image features in the second sample image features.

2. The semantic segmentation method according to claim 1, characterized in that: The method for acquiring the at least two second sample sub-image features comprises: Using an area in the sample query image except an area where the second sample category labeling result is located as a sample background area of ​​the sample query image; Acquire the sample background image feature corresponding to the sample background area among the second sample image features; The sample background image features are clustered based on the distances between features in the sample background image features to obtain the at least two second sample sub-image features.

3. The semantic segmentation method according to claim 1, characterized in that: The process of determining the semantic segmentation result of the first sample includes: Inputting the first sample image feature and the first sample category labeling result into the first branch network; Based on the first branch network, a global average pooling process is performed on the first sample image feature and the first sample category annotation result to obtain a sample foreground prototype corresponding to a sample foreground area of ​​the sample support image; the sample foreground area of ​​the sample support image is the area where the first sample category annotation result is located; Based on the first branch network, the sample foreground prototype and the second sample image feature are fused to obtain a first sample fusion feature; The first sample fusion features are encoded based on the first branch network to obtain the first sample semantic segmentation result.

4. The semantic segmentation method according to claim 3, characterized in that: The fusing the sample foreground prototype and the second sample image feature based on the first branch network to obtain a first sample fusion feature includes: Expanding the size of the sample foreground prototype to the size of the second sample image feature based on the first branch network to obtain an expanded sample foreground prototype; The expanded sample foreground prototype and the second sample image feature are fused based on the first branch network to obtain the first sample fusion feature.

5. The semantic segmentation method according to claim 1, characterized in that: The process of determining the semantic segmentation result of the second sample includes: Inputting the at least two second sample sub-image features into the second branch network; Performing global average pooling processing on the at least two second sample sub-image features and the second sample category labeling result based on the second branch network to obtain sample background prototypes corresponding to the at least two second sample sub-image features respectively; Based on the second branch network, the at least two second sample sub-image features and the sample background prototypes corresponding to the at least two second sample sub-image features are fused to obtain a second sample fusion feature; The second sample fusion features are encoded based on the second branch network to obtain the second sample semantic segmentation result.

6. The semantic segmentation method according to claim 5, characterized in that: The fusing the at least two second sample sub-image features and the sample background prototypes corresponding to the at least two second sample sub-image features based on the second branch network to obtain a second sample fusion feature includes: Expanding the size of each sample background prototype to the size of the corresponding second sample sub-image feature based on the second branch network to obtain an expanded sample background prototype corresponding to each sample background prototype; The expanded sample background prototype corresponding to each sample background prototype and the corresponding second sample sub-image feature are fused to obtain the second sample fusion feature.

7. A semantic segmentation device, characterized in that: The device comprises: An image acquisition module, used to acquire a support image and a query image to be segmented; An input module is used to input the supporting image and the query image into a semantic segmentation network to obtain a semantic segmentation result of the query image; the supporting image is annotated with a category annotation result, and the input module includes: a foreground prototype acquisition unit, used to perform global average pooling processing on the first image feature of the supporting image and the category annotation result based on the semantic segmentation network to obtain a foreground prototype corresponding to the foreground area of ​​the supporting image; the foreground area is the area where the category annotation result is located; a fusion feature acquisition unit, used to fuse the foreground prototype and the second image feature of the query image based on the semantic segmentation network to obtain a fusion feature; a semantic segmentation result determination unit, used to encode the fusion feature based on the semantic segmentation network to obtain the semantic segmentation result; Among them, the semantic segmentation network is obtained by updating the network parameters in the preset semantic segmentation network based on the target loss value, the target loss value is determined based on the first loss value and the second loss value, the first loss value is determined based on the first sample semantic segmentation result and the second sample category annotation result annotated in the sample query image, and the second loss value is determined based on the second sample semantic segmentation result and the non-category annotation result, and the non-category annotation result represents the difference between the second sample category annotation result and 1; the first sample semantic segmentation result is obtained by comparing and learning the first sample image features of the sample support image annotated with the first sample category annotation result and the second sample image features of the sample query image based on the first branch network in the preset semantic segmentation network; the second sample semantic segmentation result is obtained by comparing and learning at least two second sample sub-image features based on the second branch network in the preset semantic segmentation network, and the at least two second sample sub-image features are obtained by clustering the sample background image features in the second sample image features.

8. The semantic segmentation device according to claim 7, characterized in that: The device also includes: A sample background region determination module, configured to use a region in the sample query image, excluding a region where the second sample category annotation result is located, as a sample background region of the sample query image; A background image feature acquisition module, used to acquire the sample background image features corresponding to the sample background area in the second sample image features; The clustering module is used to cluster the sample background image features based on the distances between the features in the sample background image features to obtain the at least two second sample sub-image features.

9. The semantic segmentation device according to claim 7, characterized in that: The device also includes: A first input unit, used to input the first sample image feature and the first sample category labeling result into the first branch network; a sample foreground prototype acquisition unit, configured to perform global average pooling processing on the first sample image feature and the first sample category annotation result based on the first branch network to obtain a sample foreground prototype corresponding to a sample foreground area of ​​the sample support image; the sample foreground area of ​​the sample support image is an area where the first sample category annotation result is located; A first sample fusion feature acquisition unit, configured to obtain a first sample fusion feature by fusing the sample foreground prototype and the second sample image feature based on the first branch network; The first encoding unit is used to encode the first sample fusion feature based on the first branch network to obtain the first sample semantic segmentation result.

10. The semantic segmentation device according to claim 9, characterized in that: The first sample fusion feature acquisition unit includes: A first expansion subunit, configured to expand the size of the sample foreground prototype to the size of the second sample image feature based on the first branch network to obtain an expanded sample foreground prototype; The first fusion subunit is used to fuse the expanded sample foreground prototype and the second sample image feature based on the first branch network to obtain the first sample fusion feature.

11. The semantic segmentation device according to claim 7, characterized in that: The device also includes: A second input unit, used for inputting the at least two second sample sub-image features into the second branch network; a sample background prototype acquisition unit, configured to perform global average pooling processing on the at least two second sample sub-image features and the second sample category labeling result based on the second branch network to obtain sample background prototypes corresponding to the at least two second sample sub-image features respectively; A second sample fusion feature acquisition unit is used to fuse the at least two second sample sub-image features and the sample background prototypes corresponding to the at least two second sample sub-image features respectively based on the second branch network to obtain a second sample fusion feature; The second encoding unit is used to encode the second sample fusion feature based on the second branch network to obtain the second sample semantic segmentation result.

12. The semantic segmentation device according to claim 11, characterized in that: The second sample fusion feature acquisition unit includes: A second expansion subunit is used to expand the size of each sample background prototype to the size of the corresponding second sample sub-image feature based on the second branch network, so as to obtain an expanded sample background prototype corresponding to each sample background prototype; The second fusion subunit is used to fuse the extended sample background prototype corresponding to each sample background prototype and the corresponding second sample sub-image feature to obtain the second sample fusion feature.

13. An electronic device for semantic segmentation, characterized in that: The electronic device includes a processor and a memory, wherein at least one instruction or at least one program is stored in the memory, and the at least one instruction or at least one program is loaded and executed by the processor to implement the semantic segmentation method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the semantic segmentation method as described in any one of claims 1 to 6.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the semantic segmentation method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Small sample image semantic segmentation method based on hybrid model

    CN111583284A

  • Semantic segmentation model training method, device and equipment based on weak supervision

    CN112085739A

  • Semantic segmentation method and device based on few samples, electronic equipment and storage medium

    CN112150471A

  • Semantic segmentation method and device based on few samples, electronic equipment and storage medium

    CN112464943A

  • Semantic segmentation network training and image semantic segmentation method, device and equipment

    CN113420827A