A semantic segmentation method, apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a semantic segmentation method, device, electronic device and storage medium, the method comprising: obtaining a set of images to be processed, extracting feature maps of each image to be processed in the set of images to be processed, and obtaining a preliminary class activation map of the feature map based on a binary cross entropy loss value. Based on the feature map and the preliminary class activation map corresponding to each category, a category feature map is obtained. The loss value of the single-label multi-category classification result of the category feature map is obtained, and based on the binary cross entropy loss value and the loss value of the single-label multi-category classification result, a class reactivation map of the feature map is obtained, and based on the class reactivation map, a pseudo label of each pixel in the image to be processed is obtained. The method of the exemplary embodiment of the present disclosure can save annotation costs, improve image segmentation performance, and solve the problem that the image category cannot be associated with the corresponding area of ​​the feature information, resulting in inaccurate image segmentation.

Description

Technical Field

[0001] The present disclosure relates to the technical field of industrial visual measurement, and in particular to a semantic segmentation method, device, electronic device and storage medium. Background Art

[0002] The goal of semantic segmentation is to assign a corresponding semantic category to each pixel in an image. It is one of the most important tasks in the field of computer vision and is widely used in tasks such as image editing and scene understanding. In the field of industrial visual measurement, multi-target measurement of complex parts is a hot application area. Currently, deep learning methods are mainly used to identify part structures, and then details are extracted to obtain scenes suitable for complex structures and high-precision measurements.

[0003] Existing semantic segmentation networks tend to only focus on a certain feature of an object, while weakening the extraction of other features. Therefore, for image areas where feature information is sparsely distributed, points belonging to that category may be assigned a lower category weight and excluded from the segmentation process; and in image areas where multi-category feature information overlaps, points of different categories will be mistakenly assigned higher weights, and different categories will be mistakenly divided together during the segmentation process. Therefore, the image category cannot be associated with the area corresponding to the feature information, resulting in inaccurate image segmentation. Summary of the invention

[0004] The present disclosure provides a semantic segmentation method, device, electronic device and storage medium.

[0005] According to a first aspect of the present disclosure, a semantic segmentation method is provided, the method comprising:

[0006] Get the image set to be processed;

[0007] Extracting a feature map of each image to be processed in the image set to be processed, and obtaining a preliminary class activation map of the feature map based on a binary cross entropy loss value;

[0008] Obtaining a category feature map based on the feature map and the preliminary class activation map corresponding to each category;

[0009] Obtaining a loss value of a single-label multi-category classification result of the category feature map;

[0010] Based on the binary cross entropy loss value and the loss value of the single-label multi-category classification result, a class reactivation map of the feature map is obtained, and based on the class reactivation map, a pseudo label of each pixel in the image to be processed is obtained.

[0011] According to a second aspect of the present disclosure, a semantic segmentation device is provided, characterized in that the device comprises:

[0012] A data acquisition module, used for acquiring an image to be processed;

[0013] A data processing module, used to extract the feature map of each image to be processed in the image set to be processed, and obtain a preliminary class activation map of the feature map based on a binary cross entropy loss value; obtain a category feature map based on the feature map and the preliminary class activation map corresponding to each category; and obtain a loss value of a single-label multi-category classification result of the category feature map;

[0014] The data processing module is also used to obtain a class reactivation map of the feature map based on the binary cross entropy loss value and the loss value of the single-label multi-category classification result, and obtain a pseudo label for each pixel in the image to be processed based on the class reactivation map.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0017] The semantic segmentation method, device, electronic device and storage medium provided by the embodiments of the present disclosure obtain a feature map of the image to be processed, and obtain a preliminary class activation map of the feature map based on the binary cross entropy loss value. A feature map related to a single category is obtained by fusing the class activation map with the feature map, and a class reactivation map based on single-label multi-category classification is constructed to obtain a new seed region, wherein the seed region is used as a new and more accurate pixel-level pseudo-label for full supervision training. More accurate pixel-level pseudo-labels can reduce the dependence on manually annotated labels during the training of semantic segmentation models, thereby saving manpower and time costs. At the same time, the class reactivation map can accurately reflect the mapping relationship between each pixel in the image and the category, realize the one-to-one correspondence between the category and the pixel, expand the area of ​​interest of the deep network, and enable the model to better divide the categories corresponding to the pixels in the area with sparse feature information, thereby improving the accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A schematic diagram showing an example system in which various methods described herein may be implemented according to an exemplary embodiment of the present disclosure;

[0020] Figure 2A flowchart of a semantic segmentation method according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 3 A schematic diagram showing the principle of converting a feature graph into a class reactivation graph according to an exemplary embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram showing the principle of obtaining pixel-level pseudo labels after correction based on a class reactivation map according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 5 A schematic diagram showing the experimental effect of pseudo-labeling generated by the semantic segmentation method provided by the present disclosure for bearing images;

[0024] Figure 6 A segmentation result diagram of a bearing image generated using the semantic segmentation method provided by the present disclosure is shown;

[0025] Figure 7 A schematic block diagram of functional modules of a semantic segmentation device provided by an exemplary embodiment of the present disclosure;

[0026] Figure 8 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown;

[0027] Fig. 9 The present invention is a block diagram of an electronic device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that the various steps described in the method implementation of the present disclosure can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between the multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] Before introducing the embodiments of the present disclosure, the following definitions are given for the terms involved in the embodiments of the present disclosure:

[0034] Weakly supervised learning, by using weakly labeled images such as label box level, marked point or line level, image level, etc. to automatically extract the fully supervised label or mask of the image, and then obtain a high-performance semantic segmentation model through full supervision training.

[0035] PASCAL VOC 2012 dataset,The PASCAL VOC 2012 dataset is one of the most commonly used datasets in computer vision research and has become one of the standard datasets for evaluating the performance of computer vision algorithms.

[0036] Deeplab V2, Deeplab V2 is a semantic segmentation algorithm based on deep learning. It uses deep convolutional neural networks to perform pixel-level semantic segmentation and is mainly used for image segmentation tasks in the field of computer vision.

[0037] UperNet-Swin Platform,UperNet-Swin is a computer vision platform that combines the two technologies,UperNet and SwinTransformer, to form a powerful image semantic segmentation model.

[0038] Deep Residual Network (ResNet), ResNet50 is a deep residual network model with 50 layers of convolutional neural network. It was proposed by researchers from Microsoft Research and is one of the ResNet series models. It can be used to quickly build and train computer vision models such as image classification, target detection, and semantic segmentation.

[0039] VGG model, the VGG model is a deep convolutional neural network proposed by the Visual Geometry Group in 2014. The VGG model improves network performance by using a very small convolution kernel size (3×3) and a network structure of different depths.

[0040] Masking is a common operation in deep learning. It is equivalent to covering the original tensor with a mask to shield or select some specific elements. Therefore, it is often used to construct filters for tensors.

[0041] Iterative Refinement Network (IRN) model, the IRN model is a deep learning model for image generation tasks. It was proposed by the University of Washington and Facebook AI research team in 2017. The IRN model adopts the framework of generative adversarial networks, and its goal is to learn to generate high-quality images that match given conditions. Unlike traditional generative adversarial network models, the IRN model uses an iterative approach to generate images, and each iterative step further improves the generated image.

[0042] Adversarial Class Activation Map (AdvCAM),AdvCAM is a method for explaining adversarial examples. It is an extension of Class Activation Mapping (CAM).

[0043] Binary cross entropy loss function,The binary cross entropy loss function is a common loss function for binary classification tasks, which is usually used to measure the prediction error of the model when predicting binary classification problems.

[0044] Cross entropy loss function,The cross entropy loss function is a commonly used neural network loss function, which is usually used in classification problems. It is proposed based on the concept of information entropy theory and is used to measure the gap between the model prediction results and the true label.

[0045] Global Average Pooling (GAP) is a pooling method commonly used in convolutional neural networks. It averages each channel of all feature maps in the channel dimension to obtain a value, thereby compressing the entire feature map into a vector.

[0046] Sigmoid activation function, Sigmoid function is often used as an activation function of neural networks to map variables between 0 and 1.

[0047] ReLu activation function, ReLu (Rectified Linear Unit) is a nonlinear activation function, which is one of the most commonly used activation functions in deep learning.

[0048] The softmax activation function is a commonly used neural network activation function, which is often used in multi-classification problems. It takes a vector as input and maps it to a probability distribution, where each element represents the probability that the input belongs to each category.

[0049] Ubuntu, Ubuntu is a Linux operating system based on desktop applications.

[0050] Tenser Flow-gpu, TensorFlow-GPU is a Python-based deep learning framework that is highly optimized for GPUs. It can run on GPUs, greatly speeding up the training of deep learning models.

[0051] Compute Unified Device Architecture (CUDA) is a general-purpose parallel computing architecture launched by NVIDIA. It aims to achieve more efficient parallel computing with cheaper device resources.

[0052] MSCOCO dataset,MSCOCO dataset is a large image dataset developed and maintained by Microsoft. The tasks of frequency aggregation include recognition, segmentation, and detection.

[0053] Softmax regression, the softmax regression method is used for multi-category classification. When the input data has multiple categories, softmax regression is used to estimate the probability that the input data belongs to each category.

[0054] In the field of industrial visual measurement, multi-target measurement of complex parts is a hot application area. At present, deep learning methods are mainly used to identify part structures, and then extract details to obtain scenes suitable for complex structures and high-precision measurements. In addition, in complex motion systems, such as autonomous robots and unmanned vehicles, the main technology for spatial modeling is based on the feature information and boundaries of 2D images. These fields have a basic demand for graphic segmentation technology.

[0055] In the current field of image segmentation, the semantic segmentation model of full supervision training has been able to achieve good results. However, since the semantic segmentation model of full supervision training requires pixel-level annotation, which requires a lot of manpower and time costs, it has become the main obstacle to its widespread application. How to reduce the cost of manual annotation and provide comparable detection performance is the focus of current research. Therefore, in order to save annotation costs, improve image segmentation performance and obtain better segmentation effects, the embodiments of the present disclosure first provide a semantic segmentation method, which is applied in application fields such as image segmentation, object recognition and scene understanding.

[0056] Figure 1 Schematic diagram of an example system in which various methods described herein may be implemented according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the system 100 of the exemplary embodiment of the present disclosure may include: a user device 110 , a computing device 120 , and a data storage system 130 .

[0057] like Figure 1 As shown, the user equipment 110 can communicate with the computing device 120 through a communication network. The communication network can be a wired communication network or a wireless communication network. The wired communication network can be a communication network based on power line carrier technology, and the wireless communication network can be a local wireless network or a wide area wireless network. The local wireless network can be a WIFI wireless network, a Zigbee wireless network, a mobile communication network, a satellite communication network, etc.

[0058] like Figure 1As shown, the user device 110 may include a computer, a mobile phone, or an intelligent terminal such as an information processing center. The user device 110 may serve as the initiator of the model training operation or the feature map extraction operation and initiate a request to the computing device 120. The computing device 120 may be a server with data processing functions such as a cloud server, a network server, an application server, and a management server, for implementing the training method and the generation method. A deep learning processor may be configured in the server, and the deep learning processor may be a neuron of a single-core deep learning processor (Deep Learning Processor-Singlecore, abbreviated as DLP-S) or a multi-core deep learning processor (Deep Learning Processor-Multicore, abbreviated as DLP-M). DLP-M is a multi-core extension based on DLP-S, and multiple DLP-S are interconnected, multicast, and inter-core synchronization protocols are used to communicate between cores through the on-chip network (Network-on-chip, abbreviated as Noc) to complete deep learning tasks and expression image generation tasks.

[0059] like Figure 1 As shown, the above-mentioned data storage system 130 can be a general term, including local storage and databases storing historical data. The database can be on the computing device 120, on other network servers, or on the data storage system 130. The data storage system 130 can be separate from the computing device 120 or integrated in the computing device 120. The data storage system 130 can not only input data uploaded by the user device, but also store program instructions, neuron data, weight data, etc. These neuron data and weight data can be trained data or untrained data, which is determined by the method implemented by the computing device 120. In addition, the data storage system 130 can also store the processing results obtained by the computing device 120 (such as feature maps, preliminary class activation maps based on binary cross entropy loss, class reactivation maps obtained based on single-label multi-class classification loss, pixel-level pseudo labels, intermediate processing results or image quality) into the data storage system.

[0060] In practical applications, such as Figure 1As shown, the user device 110 may have a data set acquisition function, so that the user device can not only initiate a request to the execution device through the interactive interface, but also collect or build a training image data set, and send the image data set to the computing device 120 through the communication network. Based on this, when the computing device 120 implements the training method, the sample images used can not only be obtained from the data storage system 130, but also from the user device 110 through the communication network. In addition, when the computing device 120 implements the generation method, the pixel-level pseudo-labels generated by it can not only be fed back to the user device 110 through the communication network, but also be stored in the data storage system 130 as sample data for further training the model.

[0061] For example, Figure 1 As shown, the user device 110 may have a display interface, and image processing software such as Photoshop, MATLAB, OpenCV, Metahuman Creator (Unreal Engine Face Pincher) may be installed locally on the user device. The user device 110 may respond to user operations, use these image processing software to pre-process the image data set offline, and then upload it to the computing device 120. Photoshop, MATLAB, OpenCV, Metahuman Creator (Unreal Engine Face Pincher) and other image processing software may also be installed in the computing device 120. The user may respond to user operations, remotely log in to the execution device, and use these image processing software to process the image data set.

[0062] Figure 2 A flowchart of a semantic segmentation method according to an exemplary embodiment of the present disclosure is shown. Figure 2 As shown, the method may include the following steps:

[0063] Step S210: Obtain a set of images to be processed. The set of images to be processed here refers to any picture data that needs to be semantically segmented, including but not limited to public data sets and special part photo data.

[0064] In an optional manner, the image set to be processed may be an image dataset, which may be used as a training dataset and a test dataset for the semantic segmentation method of the exemplary embodiment of the present disclosure, wherein the training dataset is used to train a class reactivation graph model for weakly supervised semantic segmentation, and the test dataset is used to cross-validate the model and complete hyperparameter tuning. In practical applications, the training dataset and the test dataset maintain the same preprocessing logic.

[0065] Exemplarily, the preprocessing steps of the image dataset include but are not limited to:

[0066] According to the actual situation, the image dataset is selectively subjected to random horizontal and vertical scale cropping, random image flipping, or random image brightness change operations.

[0067] The processed image dataset is uniformly output to a fixed size.

[0068] The processed image dataset is then randomly divided into a training dataset and a test dataset, which are used for training and testing of the class reactivation graph model for weakly supervised semantic segmentation, respectively.

[0069] Step S220: extracting a feature map of each image to be processed in the image set to be processed, and obtaining a preliminary class activation map of the feature map based on the binary cross entropy loss value.

[0070] After the image to be processed is preprocessed, feature map extraction begins. Any image x is input into the feature extraction backbone network f to obtain the corresponding multi-channel H×W×C feature map f(x). The feature extraction backbone network here can be a VGG model or a ResNet-50 model.

[0071] The process of obtaining the preliminary class activation map of the feature map based on the binary cross entropy loss value can be a process of calculating the sensitivity of a certain category to each position of the feature map through back propagation. The sensitivity here is the contribution of the corresponding position to the prediction result. The specific steps include:

[0072] First, a global average pooling operation is performed on the feature map.

[0073] Exemplarily, the calculation process of global average pooling is as follows: for the input feature map, the elements of each channel are summed to obtain the sum of the channels, and then the sum of the channels is averaged to obtain the average value of each channel, and finally the average value of each channel is connected into a vector, thereby compressing the entire H×W×C feature map f(x) into a 1×1×C feature vector as the output result after pooling.

[0074] The benefit of global average pooling is that it can compress feature maps of different sizes into vectors of the same size, so that inputs of different sizes can be processed in the fully connected layer. In addition, compared with traditional pooling methods, global average pooling can better preserve the information of feature maps and avoid information loss.

[0075] Then, the 1×1×C feature vector is input into the fully connected layer F1, and the sigmoid activation function is used after each output unit of F1 to form a multi-label classification score, so as to judge whether each category appears in the image separately. This step can be expressed by the following formula (1):

[0076] s=σ(F1(P(f(x)))) (1)

[0077] Among them, σ represents the sigmoid activation function, P represents the global average pooling, f(x) represents the feature map, and F1 represents the fully connected layer.

[0078] Calculate the binary cross entropy loss and optimize the fully connected layer F1. The weights of the fully connected layer are used to generate preliminary class activation maps. This step can be expressed by the following formula (2):

[0079]

[0080] Among them, s[n] represents the classification score of the nth class, N represents the total number of foregrounds in the dataset, and l[n]∈{0,1} represents the image-level label of the nth class, which is 1 if the nth class exists in the corresponding image, otherwise it is 0.

[0081] Calculate the preliminary class activation map based on the binary cross entropy loss. For the feature map f(x), in order to calculate its class activation map of the nth class, first add and nonlinearly activate according to the classification weights of each channel of the nth class in the fully connected layer F1. In an optional method, the nonlinear activation can use the ReLu activation function. This step can be expressed by the following formula (3):

[0082]

[0083] in, It represents the classification weight of the nth class in the fully connected layer F1, and Act represents the nonlinear activation function.

[0084] Then use the maximum value normalization. This step can be expressed by the following formula (4):

[0085]

[0086] Among them, C n (x) represents the preliminary class activation map.

[0087] For the preliminary class activation map, the larger the value of the category score, the darker the color, and the more important the corresponding area position.

[0088] Step S230: Based on the feature map and the preliminary class activation map corresponding to each category, obtain the category feature map. Re-assign the weights of the points on the feature map corresponding to each category through the preliminary class activation map. For the nth category, the preliminary class activation map C obtained in the previous step can be n (x) acts as a spatial soft mask and acts on the original feature map f(x) to reintegrate the feature map f related to category n n (x). The original feature map f(x) is integrated with the class activation map of the nth class to obtain the feature map related to class n. This step can be expressed by the following formula (5):

[0089]

[0090] in, represents element-wise multiplication, f i represents the sub-feature map of the i-th channel in the original feature map f(x), the original feature map f(x) has a total of I channels, […] represents the channel-by-channel concatenation feature map, C n (x) represents the preliminary class activation map, f n (x) represents the category-related feature map of the nth category.

[0091] The purpose of redistributing weights of points on the feature map corresponding to each category through the preliminary class activation map is to strengthen the response of the neural network to specific categories, thereby improving the performance and accuracy of the neural network.

[0092] In the prior art, a global average pooling layer is used to convert a feature map into a feature vector, and the feature vector is used as the input of the classifier. However, this method ignores the position information of each point in the feature map, which may lead to a decrease in the performance of the classifier. To overcome this problem, a preliminary class activation map can be used to visualize the feature map corresponding to each category, and the weights can be redistributed according to the preliminary class activation map. The class activation map method obtains the pixel-level weights on the feature map corresponding to each category by multiplying the weights of the classifier and the feature map, and then superimposes these weights on the input image to obtain the class activation map of the category. In the class activation map method, the pixel-level weights of the class activation map reflect the degree of response of the neural network to a specific category, which can be used to redistribute the weights of each point on the feature map, thereby strengthening the response of the neural network to the category and improving the performance and accuracy of the classifier.

[0093] Step S240: Obtain the loss value of the single-label multi-category classification result of the category feature map.

[0094] For the category-related feature map f of the nth category n (x) first undergoes global average pooling, then is input into the fully connected layer F2, and through the softmax activation function, the weights of each channel sub-feature map in the category-related feature map are learned only for category n. Finally, these weights are used to re-weight the channels of the category-related feature map to obtain a new category score. Due to the normalization property of softmax, the activation of category n can be increased while suppressing the activation of other irrelevant categories. This step can be expressed by the following formula (6):

[0095] s′ n =softmax(F2(P(f n (x)))) (6)

[0096] Among them, s n ′[n] represents s′ n The nth element value in is the classification score of the nth class, F2 represents the fully connected layer, P represents the global average pooling, and f n (x) represents the category-related feature map of the nth category.

[0097] The loss of single-label multi-category classification is calculated by the cross entropy loss function. This step can be expressed by the following formula (7):

[0098]

[0099] Among them, l[n]∈{0,1} represents the image-level label of the nth class. If the nth class exists in the corresponding image, it is 1, otherwise it is 0. There are a total of N classes in the image. s′ n [n] represents s′ n The nth element value in is the classification score of the nth category.

[0100] Step S250: Based on the binary cross entropy loss value and the loss value of the single-label multi-category classification result, a class reactivation map of the feature map is obtained, and based on the class reactivation map, a pseudo label of each pixel in the image to be processed is obtained.

[0101] First, the overall loss value is calculated based on the binary cross entropy loss value and the loss value of the single-label multi-class classification result, and the network parameters are optimized by gradient descent. The two loss functions L1 and L2 are added, where L2 is multiplied by the μ hyperparameter to control its optimization weight. The overall loss value can be expressed by the following formula (8):

[0102] L C =L1 + μL2 (8)

[0103] Among them, L C Represents the overall loss value, L1 represents the binary cross entropy loss function, L2 represents the cross entropy loss function, and μ represents the hyperparameter.

[0104] By calculating the gradient of the overall loss value with respect to the F1 and F2 weights, and then back-propagating to optimize the F1 and F2 weights, the overall loss value is gradually reduced to achieve the purpose of training the model.

[0105] Then, the class reactivation map of the nth class is extracted from the original feature map f(x). This step uses the channel weights corresponding to the nth class output in F1 and F2. First, the original feature map f(x) is weighted according to the channel, and then it is subjected to nonlinear activation and maximum normalization to obtain the class reactivation map of the nth class. In an optional method, the nonlinear activation can use the ReLu activation function, and the calculation process can be expressed by the following formulas (9)-(11):

[0106]

[0107]

[0108]

[0109] in, represents element-wise addition, represents the weight corresponding to category n in the fully connected layer F1, represents the weight corresponding to category n in the fully connected layer F2, Act represents the nonlinear activation function, Represents the class reactivation graph for the nth class.

[0110] Figure 3 FIG. 1 is a schematic diagram showing the principle of converting a feature graph of an exemplary embodiment of the present disclosure into a class reactivation graph. Figure 3 As shown, a global average pooling operation is performed on the multi-channel H×W×C feature map f(x) to obtain a 1×1×C pooled feature vector, where the global average pooling operation is shown in step S220 above. Then, the 1×1×C feature vector is input into the fully connected layer F1, and a sigmoid activation function is used after each output unit of F1 to output the multi-label classification result.

[0111] In an optional method, six categories can be pre-set, namely: people, bicycles, cats, dogs, buses and trees. The categories and their corresponding element positions can be expressed as {people, bicycles, cats, dogs, buses, trees}. When people, cats and trees exist in the original image, the multi-label classification result can be expressed as {1,0,1,0,0,1}. The elements in the set represent the classes in the original image. If the corresponding class exists in the original image, the element corresponding to the class is output as 1. Otherwise, the corresponding element is output as 0.

[0112] According to the classification weights of each channel of the nth category in the fully connected layer F1, the nonlinear activation is performed, and then the maximum value is normalized to obtain the preliminary class activation map. Preliminary class activation map C n (x) and the sub-feature map f of the ith channel in the feature map f(x) i Perform a cross multiplication operation to obtain the feature map f related to category n n (x).

[0113] For the category-related feature map f of the nth category n (x) first undergoes global average pooling, then is input into the fully connected layer F2, and passes through the softmax activation function to output the single-label multi-category classification result.

[0114] In an optional method, the single-label multi-category classification results can be expressed as {1,0,0,0,0,0}, {0,0,1,0,0,0} and {0,0,0,0,0,1}, where {1,0,0,0,0,0} corresponds to the human category, {0,0,1,0,0,0} corresponds to the cat category, and {0,0,0,0,0,1} corresponds to the tree category.

[0115] Finally, the channel weights corresponding to the n-th class output in F1 and F2 are used First, the feature map f(x) is weighted according to the channel, and then subjected to nonlinear activation and maximum normalization to obtain a class reactivation map of the nth class. For details, please see step S250 above.

[0116] Based on the generation of class activation maps, the multi-label classification problem is converted into multiple single-label multi-category classification problems with mutual penalties between categories. The class activation maps are further integrated to form category-related class reactivation maps, so that the obtained pseudo-mask field of view has a stronger correlation with the corresponding category and a lower correlation with the non-corresponding category, which has a positive optimization effect on the image segmentation performance.

[0117] Finally, based on the class reactivation map, the pseudo labels of each pixel in the image to be processed are obtained, and the obtained pixel-level pseudo labels are used for the fully supervised training of the semantic segmentation model. The pixel-level pseudo-label obtained after correction is used as the supervision information for subsequent fully supervised image segmentation training. This step can be expressed by the following formula (12):

[0118]

[0119] in, represents the class reactivation map of the nth class, and t represents the tth step of the training process. For all N class reactivation maps, the multi-label classification scores of N classes are obtained as seed masks. The seed masks here can be used to generate seed regions as pixel-level fully supervised classification labels of the corresponding categories. The pixel-level fully supervised classification labels here are pixel-level annotations corresponding to each category, which mark which category each pixel in the image belongs to.

[0120] The image-level labels corresponding to the images to be semantically segmented are input into the determined class reactivation mapping model to obtain the corresponding pixel-level image segmentation pseudo labels. The pixel-level pseudo labels are generated based on the class reactivation map for fully supervised training of the semantic segmentation model. The image-level label here is annotated for the entire image, which is annotated with which categories the image contains. The pixel-level image segmentation pseudo labels here are pixel-level annotations generated by the determined class reactivation mapping model after using the seed mask generated by the class reactivation map, which are annotated with which category each pixel in the image belongs to.

[0121] Figure 4 FIG. 1 shows a schematic diagram of the principle of obtaining pixel-level pseudo labels after correction based on a class reactivation map according to an exemplary embodiment of the present disclosure, as shown in FIG. Figure 4 As shown, one square corresponds to one class reactivation graph Its category is n, and the corresponding training step is t. Different blocks are class reactivation maps of different training steps, where each training step is 1-t, and T represents the final training step. The reactivation maps obtained from all training steps are accumulated to obtain an accumulated class reactivation map, which is the one in formula (12). In formula (12), Represents the maximum pixel value in the accumulated class reactivation map, corresponding to Figure 4 The accumulated class reactivation map is divided by the maximum pixel value for normalization, and the normalized label is the pixel-level pseudo label.

[0122] During the model training process, an image to be processed containing a pseudo label is obtained, the image to be processed containing the pseudo label is input into a preset model, a trained preset model is obtained, and the trained preset model is used as a semantic segmentation model.

[0123] During model training, the model learns the category information of samples based on single-label multi-class classification loss to make classification predictions. However, due to the complexity of the network and insufficient training data, the model may make classification errors. These errors may cause a large gap between the model's prediction results and the true labels, thereby reducing the performance of the model.

[0124] To solve this problem, the class reactivation map technology can be used to extract pixel-level category information, and the model can be fully supervised trained with the corrected pixel-level pseudo-labels, thereby further improving the classification performance of the model. Specifically, the class reactivation map can reorganize and weight the features of the middle layer of the model to better extract and utilize category-related features. By extracting and utilizing category-related features, more accurate category information can be obtained and more accurate pixel-level pseudo-labels can be generated. During the training process, the pixel-level pseudo-labels can be used as additional label information together with the original labels to calculate the loss function, and the parameters of the model can be updated through the back-propagation algorithm, thereby further improving the classification performance of the model. Therefore, the role of obtaining pixel-level pseudo-labels after correction based on the class reactivation map is to improve the classification performance of the model.

[0125] Finally, the image to be segmented is obtained, and the image to be segmented is input into the semantic segmentation model to obtain the segmentation result of the image to be segmented.

[0126] The present disclosure proposes a semantic segmentation method, which can obtain a feature map related to a single category by fusing the class activation map with the feature map on the basis of the original class activation map, and increase the construction of a class reactivation map based on single-label multi-category classification to obtain a new seed area, wherein the seed area is used as a new and more accurate pixel-level pseudo-label for full-supervised training. More accurate pixel-level pseudo-labels can reduce the dependence on manually annotated labels during the training of semantic segmentation models, and complete image segmentation when only image-level labels are required, thereby saving manpower and time costs. At the same time, the class reactivation map can accurately reflect the mapping relationship between each pixel in the image and the category, realize the one-to-one correspondence between the category and the pixel, expand the focus area of ​​the deep network, and enable the model to better divide the categories corresponding to the pixels in the area with sparse feature information, thereby improving the accuracy of image segmentation and obtaining a semantic segmentation model with higher performance.

[0127] In one embodiment provided by the present disclosure, both the test data set and the verification data set are PASCAL VOC 2012 data sets. PASCAL VOC 2012 data set is a standard data set, which mainly has four major categories, namely people, common animals, traffic vehicles, and indoor furniture and supplies, and mainly serves three types of tasks: image classification, object detection and recognition, and image segmentation. Using this data set as a test data set and a verification data set can effectively verify the effectiveness of the semantic segmentation method of the present disclosure applied to the standard data set.

[0128] The hardware environment, network model and data set used in this embodiment are: ResNet-50 is applied on DeepLabV2 and UperNet-Swin platforms. The hardware environment is Ubuntu, Tenser Flow-gpu, CUDA10.2, and the data set is PASCAL VOC 2012.

[0129] First, a data set is obtained and preprocessed, and the method adopts step S210 described in detail above. Feature maps are extracted based on the preprocessed data set, and a preliminary class activation map of the feature map is obtained based on the binary cross entropy loss value, and the method adopts step S220 described in detail above. Secondly, based on the feature map and the preliminary class activation map corresponding to each category, a category feature map is obtained, and the method adopts step S230 described in detail above. The loss value of the single-label multi-category classification result of the category feature map is obtained, and the method adopts step S240 described in detail above. Finally, based on the binary cross entropy loss value and the loss value of the single-label multi-category classification result, the class reactivation map of the feature map is obtained, and based on the class reactivation map, the pseudo-label of each pixel in the image to be processed is obtained, and the method adopts step S250 described in detail above.

[0130] Table 1 shows the experimental results of the semantic segmentation method disclosed in the present invention applied to the PASCAL VOC 2012 dataset:

[0131] Table 1 Experimental results applied to the PASCAL VOC 2012 dataset

[0132]

[0133] As shown in Table 1, in the DeeppLabV2 platform, the recognition accuracy of the PASCAL VOC 2012 dataset based on class reactivation mapping has different improvements compared with class activation mapping, whether it is used as a verification dataset or a test dataset. Among them, there is an average improvement of 4.0% compared with the use of the ResNet-50 semantic segmentation model, an average improvement of 2.9% compared with the use of the IRN semantic segmentation model, and an average improvement of 0.8% compared with the use of the AdvCAM+IRN semantic segmentation model. It can be seen that this embodiment effectively illustrates the effectiveness of the semantic segmentation method proposed in the present disclosure.

[0134] In one embodiment provided by the present disclosure, the verification data set is the MSCOCO standard data set, the main contents of which are: object segmentation; text recognition; multiple objects per image; more than 300,000 images; more than 2 million instances; 80 categories; an average of Figure 5Targets; There are key points for 10,000 people. Using this dataset as a validation dataset can effectively verify the effectiveness of the semantic segmentation method disclosed in the present invention on a standard dataset.

[0135] The hardware environment, network model and data set used in this embodiment are: ResNet-50, IRN, AdvCAM+IRN are applied on DeepLab and UperNet-Swin platforms. The hardware environment is Ubuntu, TenserFlow-gpu, CUDA10.2, and the data set is the MSCOCO data set.

[0136] First, a data set is obtained and preprocessed, and the method adopts step S210 described in detail above. Feature maps are extracted based on the preprocessed data set, and a preliminary class activation map of the feature map is obtained based on the binary cross entropy loss value, and the method adopts step S220 described in detail above. Secondly, based on the feature map and the preliminary class activation map corresponding to each category, a category feature map is obtained, and the method adopts step S230 described in detail above. The loss value of the single-label multi-category classification result of the category feature map is obtained, and the method adopts step S240 described in detail above. Finally, based on the binary cross entropy loss value and the loss value of the single-label multi-category classification result, the class reactivation map of the feature map is obtained, and based on the class reactivation map, the pseudo-label of each pixel in the image to be processed is obtained, and the method adopts step S250 described in detail above.

[0137] Table 2 shows the experimental results of applying the semantic segmentation method disclosed in the present invention to the MSCOCO dataset:

[0138] Table 2 Experimental results applied to the MSCOCO dataset

[0139]

[0140] As shown in Table 2, in the DeeppLabV2 platform, the MSCOCO dataset is used as a verification dataset, and the recognition accuracy based on class reactivation mapping has different improvements compared to class activation mapping. Among them, there is an average improvement of 0.7% compared to the use of the ResNet-50 semantic segmentation model, an average improvement of 1.4% compared to the use of the IRN semantic segmentation model, and an average improvement of 2.0% compared to the use of the AdvCAM+IRN semantic segmentation model. It can be seen that this embodiment effectively illustrates the effectiveness of the semantic segmentation method proposed in the present disclosure.

[0141] In one embodiment provided by the present disclosure, the training data set and the test data set are self-built data sets. The main content of the data set is: use a 20-megapixel industrial camera to take pictures of various bearings, take 200 pictures, of which 100 are used for training and 100 are used for testing. The training data set is used to train the above-mentioned class reactivation graph model of weakly supervised semantic segmentation, and the test data set is used to cross-validate the model to complete hyperparameter tuning. Using this data set as the training data set and the test data set can effectively verify the effectiveness of the semantic segmentation method of the present disclosure in the field of parts detection.

[0142] In the field of parts inspection, it is necessary to perform semantic segmentation on the cross-sectional image data of the bearing to separate the outer ring, inner ring, ball, lubricant texture, noise background area, etc. of the bearing. The image dataset can be a cross-sectional image dataset of the bearing collected by a camera or a camera. The main objects include: bearings, balls, lubricant texture, shadow interference, defects and other categories. By using the image dataset to train the class reactivation mapping model for weakly supervised semantic segmentation, the image data with the above objects can be semantically segmented.

[0143] First, a data set is obtained and preprocessed, and the method adopts step S210 described in detail above. Then, feature map extraction is performed based on the preprocessed data set, and a preliminary class activation map of the feature map is obtained based on the binary cross entropy loss value, and the method adopts step S220 described in detail above. Secondly, based on the feature map and the preliminary class activation map corresponding to each category, a category feature map is obtained, and the method adopts step S230 described in detail above. The loss value of the single-label multi-category classification result of the category feature map is obtained, and the method adopts step S240 described in detail above. Finally, based on the binary cross entropy loss value and the loss value of the single-label multi-category classification result, the class reactivation map of the feature map is obtained, and based on the class reactivation map, the pseudo-label of each pixel in the image to be processed is obtained, and the method adopts step S250 described in detail above.

[0144] Specifically, a global average pooling P is performed on the sub-feature map of size H×W in each channel of f(x) to obtain a pooled feature vector of 1×1×C, which is equivalent to compressing each feature map, where C represents the number of channels.

[0145] Finally, the 1×1×C pooled feature vector is input into the fully connected layer F1. During this process, the pooled feature vector and the weight matrix w1 in the fully connected layer F1 are matrix multiplied to obtain a 20-dimensional vector. Softmax regression is performed on the 20-dimensional vector so that each vector element is output in the form of probability. The sum of the probabilities of the vector elements is 1. The probability of the vector elements here is the basis for the final classification of the image.

[0146] In general, the process of building a hierarchical semantic network model is from pooling feature vectors to using probability vectors to weight feature vectors to obtain the probabilities of each category. Since the pooled feature vectors come from feature maps, W is to select the appropriate feature layer for each category and weight it. By training the weight matrix w1, the selection of feature layers will become more and more accurate. In the actual training process, since the value of W has just been initialized, the hierarchical semantic network model makes judgments based on the w1 matrix at this time, and selects the 10th, 20th, and 30th feature maps as the basis for judgment. Finally, it is judged that 84% of the image is the main axis. Since the predicted loss value is large, the w1 matrix is ​​continuously updated during the subsequent back propagation, and the feature maps such as the 99th, 100th, and 101st are gradually used as the basis for judgment. Until it can be judged that 100% of the image is the main axis, it can be considered that the hierarchical semantic network model has learned to judge the main axis and the ball bearing.

[0147] Figure 5 FIG. 4 is a schematic diagram showing the experimental effect of generating pseudo labels for bearing images using the semantic segmentation method provided by the present disclosure. Figure 5 As shown, the upper left is the test data of this embodiment. The lower left is the label obtained based on class activation mapping. As shown in the figure, the label obtained based on class activation mapping can identify the outer ring, ball and inner ring from the bearing photo, but the edge of the ball is blurred. It can be seen that the recognition effect of the label obtained based on class activation mapping is not accurate enough. The upper right is the image segmentation result without classified labels. As shown in the figure, the unclassified label can accurately identify the outer ring and part of the ball of the bearing, but the arc part of the inner ring area is missing. The lower right is the label obtained based on class reactivation mapping. As shown in the figure, the label obtained based on class reactivation mapping can clearly identify the outer ring, ball and inner ring of the bearing, and the edge identification of the corresponding area is clear. It can be seen that compared with the label obtained based on class activation mapping, the recognition effect of the label obtained based on class reactivation mapping is more accurate.

[0148] Based on the class reactivation map, pixel-level pseudo labels are generated for training the fully supervised segmentation model to obtain the segmentation structure map. Figure 6 FIG. 4 shows a segmentation result diagram of a bearing image generated using the semantic segmentation method provided by the present disclosure, as shown in FIG. Figure 6 As shown, the left side is the original image data, the middle is a schematic diagram of the experimental effect of bearing component segmentation obtained based on class activation mapping, and the right side is a schematic diagram of the experimental effect of bearing component segmentation obtained based on class reactivation mapping. Figure 6 As shown in the figure, the component segmentation based on class reactivation mapping can extract the rolling ball part in the middle of the ring, while the segmentation based on class activation mapping cannot analyze the rolling ball part. It can be seen that class reactivation mapping can better extract components with large structural differences and has richer semantic feature extraction capabilities.

[0149] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a semantic segmentation system, including a first obtaining unit, a first constructing unit, a first processing unit, a second obtaining unit, and a second processing unit.

[0150] The first acquisition unit takes the image as input and obtains the feature map through the feature extraction backbone network. The first construction unit takes the feature map as input, outputs the image-level classification score through the fully connected layer, and optimizes the fully connected layer through the image-level label drive. The first processing unit takes the feature map as input, weights the feature map through the weight coefficient in the optimized fully connected layer, outputs the class reactivation map as the seed area, and generates pixel-level pseudo labels based on it for full supervision training of the semantic segmentation model.

[0151] The second obtaining unit obtains the image to be segmented, that is, the test image. The second processing unit inputs the obtained test image to be segmented into the trained semantic segmentation model, performs semantic segmentation and complete shape fitting on the image to be segmented, and outputs the semantic segmentation result of the image to be segmented.

[0152] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a semantic segmentation device, which may be a server or a chip applied to a server. Figure 7 This is a schematic block diagram of the functional modules of a semantic segmentation device provided by an exemplary embodiment of the present disclosure. Figure 7 As shown, the semantic segmentation device 700 includes: a data acquisition module 710 and a data processing module 720.

[0153] A data acquisition module 710 is used to acquire an image to be processed;

[0154] The data processing module 720 is used to extract the feature map of each image to be processed in the image set to be processed, and obtain a preliminary class activation map of the feature map based on the binary cross entropy loss value; obtain a category feature map based on the feature map and the preliminary class activation map corresponding to each category; and obtain a loss value of a single-label multi-category classification result of the category feature map;

[0155] The data processing module 720 is also used to obtain a class reactivation map of the feature map based on the binary cross entropy loss value and the loss value of the single-label multi-category classification result, and obtain a pseudo label of each pixel in the image to be processed based on the class reactivation map.

[0156] The data processing module 720 is further used to obtain a multi-label classification result based on the feature map, and the multi-label classification result is:

[0157] s=σ(F1(P(f(x))))

[0158] Among them, σ represents the sigmoid activation function, P represents the global average pooling, f(x) represents the feature map, and F1 represents the first fully connected layer;

[0159] Based on the multi-label classification result, the binary cross entropy loss value is calculated, and the binary cross entropy loss value is:

[0160]

[0161] Where s[n] represents the classification score of the nth class, N represents the total number of foregrounds in the dataset, l[n]∈{0,1} represents the image-level label of the nth class, which is 1 if the nth class exists in the corresponding image, otherwise it is 0;

[0162] Based on the binary cross entropy loss value, a preliminary class activation map of the feature map is obtained, wherein the preliminary class activation map is:

[0163]

[0164] in, represents the classification weight of the nth class in the fully connected layer F1, and Act represents the nonlinear activation function;

[0165]

[0166] Among them, C n (x) represents the preliminary class activation map.

[0167] The data processing module 720 is also used to obtain a category feature map, including:

[0168]

[0169] in, represents element-wise multiplication, f i represents the sub-feature map of the i-th channel in the original feature map f(x), the original feature map f(x) has a total of I channels, […] represents the channel-by-channel concatenation feature map, C n (x) represents the preliminary class activation map, f n (x) represents the category feature map of the nth category.

[0170] The data processing module 720 is also used to obtain the loss value of the single-label multi-category classification result of the category feature map, including:

[0171] s′ n =softmax(F2(P(f n (x))))

[0172] Among them, s n ′[n] represents s′ nThe nth element value in is the classification score of the nth class, F2 represents the second fully connected layer, P represents the global average pooling, and f n (x) represents the category feature map of the nth category.

[0173] The loss value of the single-label multi-category classification result is:

[0174]

[0175] Among them, l[n]∈{0,1} represents the image-level label of the nth class. If the nth class exists in the corresponding image, it is 1, otherwise it is 0. There are a total of N classes in the image. n ′[n] represents s′ n The nth element value in is the classification score of the nth category.

[0176] The data processing module 720 is further used to obtain an overall loss value based on the binary cross entropy loss value and the loss value of the single-label multi-class classification result;

[0177] Optimize the weights of the first fully connected layer and the second fully connected layer according to the overall loss value, and calculate the channel weights:

[0178]

[0179] in, represents element-wise addition, represents the weight corresponding to category n in the first fully connected layer, represents the weight corresponding to category n in the second fully connected layer, represents the channel weight;

[0180] Based on the channel weights and the feature map, the class reactivation map is obtained:

[0181]

[0182]

[0183] in, represents the channel weight, Act represents the nonlinear activation function, f(x) represents the feature map, Represents the class reactivation graph for the nth class.

[0184] The data processing module 720 is further used to obtain an image to be processed containing the pseudo-label; input the image to be processed containing the pseudo-label into a preset model to obtain a trained preset model, and use the trained preset model as a semantic segmentation model. Obtain an image to be segmented, input the image to be segmented into the semantic segmentation model, and obtain a segmentation result of the image to be segmented.

[0185] Figure 8 Schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown. Figure 8 As shown, the chip 800 includes one or more (including two) processors 801 and a communication interface 802. The communication interface 802 can support the server to perform the data sending and receiving steps in the above method, and the processor 801 can support the server to perform the data processing steps in the above method.

[0186] Optional, such as Figure 8 As shown, the chip 800 also includes a memory 803, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0187] In some embodiments, Figure 8 As shown, the processor 801 performs corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 801 controls the processing operations of any one of the terminal devices, and the processor may also be called a central processing unit (CPU). The memory 803 may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory 803 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Figure 8 Various buses are labeled as bus system 804 .

[0188] The method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above-mentioned method in conjunction with its hardware.

[0189] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0190] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0191] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0192] refer to Fig. 9, a block diagram of an electronic device 900 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0193] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0194] A plurality of components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 may be any type of device capable of inputting information to the electronic device 900, and the input unit 906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 may include, but is not limited to, a disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0195] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (PU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above. Each of the methods described above may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909.

[0196] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0197] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0198] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0199] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0200] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0201] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0202] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0203] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A semantic segmentation method, characterized in that: The method comprises: Get the image set to be processed; Extract the feature map of each image to be processed in the image set to be processed, and obtain a preliminary class activation map of the feature map based on the binary cross entropy loss value; wherein, based on the feature map, obtain a multi-label classification result, and the multi-label classification result is: s=σ(F1(P(f(x)))) Among them, σ represents the sigmoid activation function, P represents the global average pooling, f(x) represents the feature map, and F1 represents the first fully connected layer; Based on the multi-label classification result, the binary cross entropy loss value is calculated, and the binary cross entropy loss value is: Where s[n] represents the classification score of the nth class, N represents the total number of foregrounds in the dataset, l[n]∈{0,1} represents the image-level label of the nth class, which is 1 if the nth class exists in the corresponding image, otherwise it is 0; Based on the binary cross entropy loss value, a preliminary class activation map of the feature map is obtained, wherein the preliminary class activation map is: in, represents the classification weight of the nth class in the fully connected layer F1, and Act represents the nonlinear activation function; Among them, C n (x) represents the preliminary class activation map; Based on the feature map and the preliminary class activation map corresponding to each category, a category feature map is obtained; the method includes: in, represents element-wise multiplication, f i represents the sub-feature map of the i-th channel in the original feature map f(x), the original feature map f(x) has a total of I channels, […] represents the channel-by-channel concatenation feature map, C n (x) represents the preliminary class activation map, f n (x) represents the category feature map of the nth category; Obtain the loss value of the single-label multi-category classification result of the category feature map; including: s′ n =softmax(F2(P(f n (x)))) Among them, s n ′[n] represents s′ n The nth element value in is the classification score of the nth class, F2 represents the second fully connected layer, P represents the global average pooling, and f n (x) represents the category feature map of the nth category; The loss value of the single-label multi-category classification result is: Among them, l[n]∈{0,1} represents the image-level label of the nth class. If the nth class exists in the corresponding image, it is 1, otherwise it is 0. There are a total of N classes in the image; s n ′[n] represents s′ n The nth element value in is the classification score of the nth category; Based on the binary cross entropy loss value and the loss value of the single-label multi-class classification result, a class reactivation map of the feature map is obtained; wherein, based on the binary cross entropy loss value and the loss value of the single-label multi-class classification result, an overall loss value is obtained; Optimize the weights of the first fully connected layer and the second fully connected layer according to the overall loss value, and calculate the channel weights: in, represents element-wise addition, represents the weight corresponding to category n in the first fully connected layer, represents the weight corresponding to category n in the second fully connected layer, represents the channel weight; Based on the channel weights and the feature map, the class reactivation map is obtained: in, represents the channel weight, Act represents the nonlinear activation function, f(x) represents the feature map, Represents the class reactivation graph for the nth class; Based on the class reactivation map, obtaining a pseudo label of each pixel in the image to be processed; Obtaining an image to be processed containing the pseudo label; The image to be processed containing the pseudo-label is input into a preset model to obtain a trained preset model, and the trained preset model is used as a semantic segmentation model.

2. The method according to claim 1, characterized in that Also includes: An image to be segmented is obtained, and the image to be segmented is input into the semantic segmentation model to obtain a segmentation result of the image to be segmented.

3. A semantic segmentation device, characterized in that: The device comprises: A data acquisition module, used for acquiring a set of images to be processed; A data processing module is used to extract the feature map of each image to be processed in the image set to be processed, and obtain a preliminary class activation map of the feature map based on the binary cross entropy loss value; wherein, based on the feature map, a multi-label classification result is obtained, and the multi-label classification result is: s=σ(F1(P(f(x)))) Among them, σ represents the sigmoid activation function, P represents the global average pooling, f(x) represents the feature map, and F1 represents the first fully connected layer; Based on the multi-label classification result, the binary cross entropy loss value is calculated, and the binary cross entropy loss value is: Where s[n] represents the classification score of the nth class, N represents the total number of foregrounds in the dataset, l[n]∈{0,1} represents the image-level label of the nth class, which is 1 if the nth class exists in the corresponding image, otherwise it is 0; Based on the binary cross entropy loss value, a preliminary class activation map of the feature map is obtained, wherein the preliminary class activation map is: in, represents the classification weight of the nth class in the fully connected layer F1, and Act represents the nonlinear activation function; Among them, C n (x) represents the preliminary class activation map; The data processing module is further used to obtain a category feature map based on the feature map and the preliminary class activation map corresponding to each category; including: in, represents element-wise multiplication, f i represents the sub-feature map of the i-th channel in the original feature map f(x), the original feature map f(x) has a total of I channels, […] represents the channel-by-channel concatenation feature map, C n (x) represents the preliminary class activation map, f n (x) represents the category feature map of the nth category; The data processing module is also used to obtain the loss value of the single-label multi-category classification result of the category feature map; including: s′ n =softmax(F2(P(f n (x)))) Among them, s n ′[n] represents s′ n The nth element value in is the classification score of the nth class, F2 represents the second fully connected layer, P represents the global average pooling, and f n (x) represents the category feature map of the nth category; The loss value of the single-label multi-category classification result is: Among them, l[n]∈{0,1} represents the image-level label of the nth class. If the nth class exists in the corresponding image, it is 1, otherwise it is 0. There are a total of N classes in the image; s n ′[n] represents s′ n The nth element value in is the classification score of the nth category; The data processing module is further used to obtain a class reactivation map of the feature map based on the binary cross entropy loss value and the loss value of the single-label multi-class classification result; wherein, based on the binary cross entropy loss value and the loss value of the single-label multi-class classification result, an overall loss value is obtained; and the weights of the first fully connected layer and the second fully connected layer are optimized according to the overall loss value, and channel weights are calculated: in, represents element-wise addition, represents the weight corresponding to category n in the first fully connected layer, represents the weight corresponding to category n in the second fully connected layer, represents the channel weight; Based on the channel weights and the feature map, the class reactivation map is obtained: in, represents the channel weight, Act represents the nonlinear activation function, f(x) represents the feature map, Represents the class reactivation graph for the nth class; The data processing module is also used to obtain the pseudo-label of each pixel in the image to be processed based on the class reactivation map; obtain the image to be processed containing the pseudo-label; input the image to be processed containing the pseudo-label into the preset model to obtain the trained preset model, and use the trained preset model as the semantic segmentation model.

4. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as claimed in any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 2.

Images