Semantic segmentation network training method, device and medium based on patch learning

Abstract

The present invention is applicable to the field of machine vision technology and provides a semantic segmentation network training method, device, electronic device and storage medium based on block patch learning. The method comprises: using a preset block patch learning mechanism to train a classification network to obtain a trained classification network, using the trained classification network to predict a category activation map of a training sample, obtaining a pseudo-label segmentation map of the training sample based on the category activation map, training a semantic segmentation network based on the training sample and the pseudo-label segmentation map to obtain a trained semantic segmentation network, thereby improving the quality of the category activation map by applying the block patch learning mechanism in the convolution block, and thereby improving the training effect of the segmentation network.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision technology, and in particular relates to a semantic segmentation network training method, device, electronic device and storage medium based on block patch learning. Background Art

[0002] As one of the basic tasks in computer vision, semantic segmentation technology is based on a given image. It accurately divides all foreground objects in the image into different categories of pixel regions and predicts the corresponding pixel-accurate segmentation masks. In recent years, with the development of deep learning technology, a semantic segmentation network based on deep learning has been trained using manually annotated datasets accurate to the pixel level. Algorithms based on such deep learning semantic segmentation technology have emerged in an endless stream and have achieved good semantic segmentation performance. However, a major pain point of this technology is that it requires a large number of manually annotated pixel-level segmentation datasets. Only by using data-driven deep learning models can good performance be achieved. This data-driven training method is based on a large number of manually annotated datasets.

[0003] The emergence of weakly supervised semantic segmentation technology can greatly alleviate the problem of requiring extensive manual labeling costs. Currently, weakly supervised semantic segmentation methods mainly include image-level annotation, point-based annotation, scribble-based annotation, and bounding box-based annotation. Among these weakly supervised annotation methods, image-level annotation offers the lowest training cost compared to the other methods. Furthermore, given the widespread application of internet technology in everyday life and work, massive amounts of image data can be easily retrieved from the internet with specific labels. This near-zero-cost image data offers significant benefits for data-driven deep learning algorithms. This patent application focuses on image-level annotation-based weakly supervised semantic segmentation technology. Currently, mainstream image-based annotation methods follow a two-stage training strategy: first, an initial class-specific activation feature map (CAM) is obtained. The corresponding class activation response mask is then mapped and refined using an image-level classification network to obtain pseudo-segmentation mask labels. Based on the generated pseudo-labels, a semantic segmentation network is then trained using a fully supervised approach. CAM technology is widely used as a method for computing initial pseudo-label masks. CAM is a multi-label classification network that computes a weighted sum of the feature maps in the last layer. However, the activation regions in CAM typically focus on computing the most discriminative local regions of objects, resulting in a failure to capture precise boundaries. This creates an inherent mismatch between image-level classification tasks and semantic segmentation training, which requires dense pixel annotation. Non-salient regions of the target object in the original image cannot be fully computed using CAM methods through standard classification networks, making it unsuitable for semantic segmentation training. Summary of the Invention

[0004] The purpose of the present invention is to provide a semantic segmentation network training method, device, electronic device and storage medium based on patch learning, aiming to solve the problem of poor training effect of semantic segmentation tasks caused by the inability of classification networks in the prior art to obtain non-salient areas of target objects.

[0005] In one aspect, the present invention provides a semantic segmentation network training method based on patch learning, the method comprising the following steps:

[0006] The classification network is trained using a preset dicing and patching learning mechanism to obtain a trained classification network, wherein the dicing and patching learning mechanism is used to perform a preset scattering operation on the output feature map of the target convolution block, input a plurality of first feature blocks obtained after performing the scattering operation into a network layer following the target convolution block to obtain a plurality of second feature blocks, perform a preset merging operation on the plurality of second feature blocks, concatenate a spliced ​​feature map obtained after performing the merging operation with the original output feature map, and then input the concatenated feature map into the classification network layer of the classification network;

[0007] Use the trained classification network to predict the category activation map of the training sample;

[0008] Obtaining a pseudo-label segmentation map of the training sample based on the class activation map;

[0009] Based on the training samples and the pseudo-label segmentation map, a semantic segmentation network is trained to obtain a trained semantic segmentation network.

[0010] Preferably, the scattering operation is used to divide the output feature map of the target convolution block into equal parts and then input the divided parts into the first shared convolution layer to obtain the multiple first feature blocks;

[0011] The merging operation is used to input the multiple second feature blocks into the second volume shared layer to obtain multiple third feature blocks, and to splice the multiple third feature blocks to obtain the spliced ​​feature map.

[0012] Preferably, the step of training the classification network using a preset dicing patch learning mechanism includes:

[0013] Using the patch learning mechanism in a current target convolutional block and training the classification network, wherein the current target convolutional block is any convolutional block in the classification network;

[0014] Determine whether the current target convolution block is the last convolution block of the classification network. If so, determine that the classification network training is completed; otherwise, use the next convolution block of the current target convolution block as the current target convolution block, and jump to the step of using the block patch learning mechanism in the current target convolution block and training the classification network.

[0015] Preferably, the step of training the classification network using a preset dicing and patching learning mechanism further includes:

[0016] The dicing patch learning mechanism is used to train a classification network that introduces a multi-branch shared network, wherein the target convolution blocks using the dicing patch learning mechanism in each network branch of the multi-branch shared network are different, and the feature maps of the target convolution blocks have the same position weights and are shared.

[0017] In another aspect, the present invention provides a semantic segmentation network training device based on patch learning, the device comprising:

[0018] A classification network training unit is configured to train the classification network using a preset dicing and patching learning mechanism to obtain a trained classification network, wherein the dicing and patching learning mechanism is configured to divide the output feature map of the target convolution block into a plurality of first feature blocks, and input each of the first feature blocks into a network layer following the target convolution block to obtain a plurality of second feature blocks, and fuse a concatenated feature map obtained by concatenating the plurality of second feature blocks with the original output feature map and input the concatenated feature map into the classification network layer, and the network parameters of the network layer preceding the target convolution block are not updated;

[0019] a class activation map generating unit, configured to use the trained classification network to predict the class activation map of the training sample;

[0020] a pseudo-label acquisition unit, configured to acquire a pseudo-label segmentation map of the training sample based on the class activation map; and

[0021] The segmentation network training unit is used to train a semantic segmentation network based on the training samples and the pseudo-label segmentation map to obtain a trained semantic segmentation network.

[0022] Preferably, the scattering operation is used to divide the output feature map of the target convolution block into equal parts and then input the divided parts into the first shared convolution layer to obtain the multiple first feature blocks;

[0023] The merging operation is used to input the multiple second feature blocks into the second volume shared layer to obtain multiple third feature blocks, and to splice the multiple third feature blocks to obtain the spliced ​​feature map.

[0024] Preferably, the classification network training unit further includes:

[0025] An implicit training unit, configured to use the patch learning mechanism in a current target convolutional block to train the classification network, wherein the current target convolutional block is any convolutional block in the classification network;

[0026] A judgment unit is used to judge whether the current target convolution block is the last convolution block of the classification network. If so, it is determined that the classification network training is completed; otherwise, the next convolution block of the current target convolution block is used as the current target convolution block, and the implicit training unit is triggered to execute the step of using the block patch learning mechanism in the current target convolution block to train the classification network.

[0027] Preferably, the classification network training unit further includes:

[0028] A fusion training unit is displayed, which is used to introduce a multi-branch shared network into a preliminarily trained classification network to train the classification network, wherein the target convolution blocks applying the block patch learning mechanism in each network branch of the multi-branch shared network are different, and the feature maps of the target convolution blocks have the same position weight sharing.

[0029] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0030] On the other hand, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0031] The present invention uses a preset dicing and patching learning mechanism to train a classification network to obtain a trained classification network, wherein the dicing and patching learning mechanism is used to perform a preset scattering operation on the output feature map of the target convolution block, input the multiple first feature blocks obtained after the scattering operation into the network layer after the target convolution block to obtain multiple second feature blocks, perform a preset merging operation on the multiple second feature blocks, cascade the spliced ​​feature map obtained after the merging operation with the original output feature map and input them into the classification network layer of the classification network, use the trained classification network to predict the category activation map of the training sample, obtain the pseudo-label segmentation map of the training sample based on the category activation map, train the semantic segmentation network based on the training sample and the pseudo-label segmentation map, and obtain a trained semantic segmentation network, thereby improving the quality of the category activation map by applying the dicing and patching learning mechanism in the convolution block, thereby improving the training effect of the segmentation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1A This is a flowchart of the implementation of the semantic segmentation network training method based on patch learning provided in the first embodiment of the present invention;

[0033] Figure 1B This is an example of the implementation process of the scattering operation and the merging operation provided in the first embodiment of the present invention;

[0034] Figure 2 This is an example diagram of the implementation process of training a classification network using a progressive implicit fusion training strategy provided in the second embodiment of the present invention;

[0035] Figure 3 This is an example diagram of the implementation process of training a classification network using an explicit fusion training strategy provided in the third embodiment of the present invention;

[0036] Figure 4A 4 is a graph showing the accuracy and recall of CAM image pixel calculations obtained after training the classification network provided in the fourth embodiment of the present invention;

[0037] Figure 4B This is a performance diagram of the CAM graph obtained after training the classification network provided by the fourth embodiment of the present invention under different thresholds;

[0038] Figure 4C This is a visualization effect diagram of a CAM graph of a portion of an input image obtained through a classification network provided by the fourth embodiment of the present invention;

[0039] Figure 4D This is a visualization diagram of the prediction results of the semantic segmentation network for part of the input image provided by the fourth embodiment of the present invention;

[0040] Figure 5 is a structural example diagram of a semantic segmentation network training device based on patch learning provided in the fifth embodiment of the present invention; and

[0041] Figure 6 It is a structural diagram of an electronic device provided in Example 6 of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] The following describes the specific implementation of the present invention in detail with reference to specific embodiments:

[0044] Example 1:

[0045] Figure 1AThe implementation process of the semantic segmentation network training method based on patch learning provided by the first embodiment of the present invention is shown. For the sake of convenience, only the part related to the embodiment of the present invention is shown, which is detailed as follows:

[0046] In step S101, a classification network is trained using a preset dicing and patching learning mechanism to obtain a trained classification network.

[0047] The embodiment of the present invention is applicable to electronic devices. In the embodiment of the present invention, the above-mentioned block patch learning mechanism deliberately breaks the feature map describing the global features of the image into several block patches, and each patch passes through the subsequent convolutional layers completely until it is spliced ​​together before entering the final classification network layer. Because the global high-level semantic features describing the entire object are deliberately decomposed and cannot be seen by the classification network, the network can only see local feature information clues related to the image object part. Therefore, more dispersed small block features contribute to the final classification prediction and obtain a category activation map (CAM map) that can better cover the entire complete object area.

[0048] Specifically, the block patch learning mechanism is used to perform a preset scattering operation on the output feature map of the target convolution block, input the multiple first feature blocks obtained after the scattering operation into the network layer after the above-mentioned target convolution block to obtain multiple second feature blocks, perform a preset merging operation on the multiple second feature blocks, and concatenate the spliced ​​feature map obtained after the merging operation with the original output feature map and input them into the last classification network layer of the classification network. During the classification network training process, the target convolution block of the block patch learning mechanism and the network layer after the target convolution block are used to update the network parameters, and the network layer before the above-mentioned target convolution block does not update the network parameters. Among them, the above-mentioned target convolution block is the convolution block using the block patch learning mechanism.

[0049] Preferably, the scattering operation is used to divide the output feature map of the target convolution block into equal parts and input the divided parts into the first shared convolution layer to obtain multiple first feature blocks. Furthermore, the merging operation is used to input multiple second feature blocks into the second shared convolution layer to obtain multiple third feature blocks, and the multiple third feature blocks are spliced ​​together to obtain a spliced ​​feature map, thereby realizing the training of the classification network through convolution sharing.

[0050] Specifically, if Figure 1B As shown, Figure 1B (a) in the equation represents a scattering operation. Figure 1B (b) in Fig. 3 represents the merging operation. In the spatial dimension of the feature map, assume that H, W, and C1 represent the height, width, and channel size of the corresponding feature map. Figure 1B F in represents the input feature map, Figure 1BThe P operation in the figure represents the equal division of the feature map in the spatial dimension. For example, if the feature map is to be divided into a K×K grid, each feature block becomes f i , whose size is Each feature block passes through a shared convolutional layer That is, the first shared convolution layer mentioned above is mainly implemented through a convolution layer with a convolution kernel size of 3x3. After the above-mentioned scattering operation, each feature block passes through all subsequent network layers completely until the final classification network layer. The classification network can only see a small area of ​​the local feature map. The multiple second feature blocks output at the end are spliced ​​back in the original order to obtain a new, complete spliced ​​feature map. Before splicing back, each feature block will first pass through another shared convolution layer. That is, the second shared convolution layer mentioned above, the convolution kernel size of the second shared convolution layer is also 3x3, and a new feature block is obtained This is the third feature block mentioned above. It should be noted here that the two shared convolutional layers mentioned above do not change the size of the feature map. The above-mentioned patch learning mechanism places more emphasis on the mining of local features within each independent feature block, and each is processed separately by the classification network and ultimately spliced ​​together. Since the global information of the image is deliberately destroyed, the classification network will be forced to enhance its perception of local target objects. In this way, non-salient target areas can also be activated. Naturally, the category activation map obtained in this way can cover more foreground areas, thereby obtaining a better initial pseudo-label segmentation map, and based on this initial pseudo-label segmentation map, the semantic segmentation network is fully supervised for training.

[0051] When using the preset dicing patch learning mechanism to train the classification network, preferably, based on the dicing patch learning mechanism, a progressive implicit fusion training strategy or an explicit fusion training strategy is used to train the classification network to enrich the training method of the classification network. Among them, when using the progressive implicit fusion training strategy to train the classification network, it is necessary to train multiple times, and when using the explicit fusion training strategy to train the classification network, it is only necessary to train once. Therefore, when training the classification network, the explicit fusion training strategy is more effective than the progressive implicit fusion training strategy, but it will occupy more existing resources. The specific implementation method of using the progressive implicit fusion training strategy to train the classification network can refer to the relevant description of implementation two, and the specific implementation method of using the explicit fusion training strategy to train the classification network can refer to the relevant description of implementation three, which will not be repeated here.

[0052] When training the classification network, the entire classification network is supervised by the multi-label classification loss function. The entire loss function is as follows:

[0053]

[0054] in, represents the loss value of the classification network, Represents the predicted label of the input image, y represents the true label of the input image, y[i] represents whether the i-th label of the input image appears in the input image, y[i]=1 represents that the i-th label of the input image appears in the input image, and y[i]=0 represents that the i-th label of the input image does not appear in the input image.

[0055] In step S102, the trained classification network is used to predict the class activation map of the training sample.

[0056] In step S103, a pseudo-label segmentation map of the training sample is obtained based on the class activation map.

[0057] In an embodiment of the present invention, the category activation map calculated by the CAM method can identify the most obvious local areas of the object, and these areas contribute the most to the final image-level classification. Therefore, this method can be used as a rough segmentation result of the target object, that is, an initial pseudo-label segmentation map. If the classification network is trained using a progressive implicit fusion training strategy, the above-mentioned initial pseudo-label segmentation map is obtained by pixel-by-pixel summing of the first segmentation mask map obtained by the network branch that does not use the segmentation patch learning mechanism in the classification network, and the second segmentation mask map obtained by the network branch that uses the segmentation patch learning mechanism. That is, the above-mentioned initial pseudo-label segmentation map is obtained by pixel-by-pixel summing of the first segmentation mask map obtained based on the original feature map and the second segmentation mask map obtained based on the above-mentioned spliced ​​feature map. The calculation method of the above-mentioned first segmentation mask map is as follows:

[0058]

[0059] in, Represents the first segmentation mask map, C represents the number of categories, k∈[1, C], represents the segmentation mask corresponding to the k-th category, The second segmentation mask can also be obtained by the above calculation method, that is, Replace with in, Represents the above-mentioned spliced ​​feature map.

[0060] If the classification network is trained using an explicit fusion training strategy, the above-mentioned initial pseudo-label segmentation map is obtained by pixel-by-pixel summing of the third segmentation mask map obtained by the network branch that does not use the segmentation patch learning mechanism in the classification network, and multiple fourth segmentation mask maps obtained by the network branch that uses the segmentation patch learning mechanism. Specifically, if the total number of network branches that use the segmentation patch learning mechanism is represented by n, the category activation map obtained using the explicit fusion training strategy contains a total of n+1 segmentation mask maps, that is, n fourth segmentation mask maps are obtained by using the n network branches that use the segmentation patch learning mechanism, and 1 third segmentation mask map is obtained by using the network branch that does not use the segmentation patch learning mechanism. The calculation method of the third and fourth segmentation mask maps also refers to the calculation method of the first segmentation mask map mentioned above, and will not be repeated here.

[0061] Because the CAM method obtains the aforementioned initial pseudo-label segmentation map from a classification network with a downsampling operation, it can only locate relatively coarse areas of the target object, which is generally difficult to meet the training requirements of the semantic segmentation network. After obtaining the aforementioned initial pseudo-label segmentation map, the class activation map or the initial pseudo-label segmentation map can be further refined to obtain a final pseudo-label segmentation map that is closer to the actual human annotation. Specifically, AffinityNet, IRNet, etc. can be used to further refine the initial pseudo-label segmentation map in the class activation map to improve the quality of the initial CAM map and obtain a final pseudo-label segmentation map that is closer to the actual human annotation. For example, SC-CAM uses AffinityNet, while MBMNet and CONTA use IRNet to further refine the obtained class activation map to obtain the final pseudo-label segmentation map. Of course, the class activation map refinement method can also be applied to the obtained initial pseudo-label segmentation map to further obtain the final pseudo-label segmentation map. This final pseudo-label segmentation map can be used as the label for training the semantic segmentation network.

[0062] In step S104, a semantic segmentation network is trained based on the training samples and the pseudo-label segmentation map to obtain a trained semantic segmentation network.

[0063] In an embodiment of the present invention, the final pseudo-label segmentation map obtained in the above manner is used as supervision information, and the above training samples are used to train the semantic segmentation network to obtain a trained semantic segmentation network.

[0064] In an embodiment of the present invention, a preset dicing and patching learning mechanism is used to train a classification network to obtain a trained classification network, wherein the dicing and patching learning mechanism is used to perform a preset scattering operation on the output feature map of the target convolution block, and input the multiple first feature blocks obtained after the scattering operation into the network layer after the target convolution block to obtain multiple second feature blocks, and perform a preset merging operation on the multiple second feature blocks. The spliced ​​feature map obtained after the merging operation is cascaded with the original output feature map and then input into the classification network layer of the classification network, and the trained classification network is used to predict the category activation map of the training sample, and the pseudo-label segmentation map of the training sample is obtained based on the category activation map. Based on the training sample and the pseudo-label segmentation map, the semantic segmentation network is trained to obtain a trained semantic segmentation network, thereby improving the quality of the category activation map by applying the dicing and patching learning mechanism in the convolution block, thereby improving the training effect of the segmentation network.

[0065] Example 2:

[0066] Figure 2 The implementation process of training a classification network using a progressive implicit fusion training strategy provided by the second embodiment of the present invention is shown. For ease of explanation, only the part related to the embodiment of the present invention is shown, which is detailed as follows:

[0067] Objects in images may require different trade-offs between local details and global features at different scales and semantic levels. To achieve this goal, this embodiment uses a progressive implicit fusion training strategy to train the classification network. Specifically, the progressive implicit fusion training strategy decomposes feature maps of different granularities into local patches and gradually trains the classification network in multiple stages. More specifically, as Figure 2 As shown in Figure 3, the low-level detail information is captured from the more stable fine-grained feature maps, and gradually turns to the high-level coarse semantic information. Figure 2 P represents the division of the feature map into equal grids in the spatial dimension. Figure 2 Here, M represents the concatenation of feature blocks in their original order, C represents the concatenation operation, cls loss represents the classification loss, classifier represents the last classification network layer of the classification network, block i represents the i-th convolution block from the shallowest layer to the deepest layer in the classification network, i∈[1,N], N represents the total number of convolution blocks in the classification network, and the classification network in the figure has undergone n stages (Stage 1-Stage n) of training.

[0068] During the training process, the patch learning mechanism is gradually transferred to higher convolution blocks. Only the target convolution block currently using the patch learning mechanism and the network layers after the target convolution block can update the parameters. The network parameters of the network layers before the target convolution block are not updated, thereby effectively reusing the local perception ability of the last granularity of information learned in the previous stage and allowing the classification network training to focus on learning the feature map of the current granularity. As the patch learning mechanism is gradually transferred to higher convolution blocks, features of different granularities are gradually implicitly fused into the network, so that the classification network has local granularity perception capabilities with different sensitivities and enables the classification network to achieve better coverage of the target object area of ​​the image. This progressive implicit fusion training strategy starts from the finer-grained bottom network layer and gradually moves to the coarser-grained top layer. The specific steps are as follows:

[0069] In step S201 , a patch learning mechanism is used in the current target convolutional block to train the classification network.

[0070] In an embodiment of the present invention, the total number of convolution blocks in the classification network can be obtained in advance, and then the network can be initialized using the ImageNet pre-trained weights. After that, a block patch learning mechanism is used in a target convolution block to train the classification network for one stage. The target convolution block can be any convolution block in the classification network, usually a low- to medium-layer convolution block. When the block patch learning mechanism is used in the current target convolution block to train the classification network, specifically, a preset scattering operation is performed on the output feature map of the target convolution block, and the multiple first feature blocks obtained after the scattering operation are input into the network layer after the target convolution block to obtain multiple second feature blocks. A preset merging operation is performed on the multiple second feature blocks, and the spliced ​​feature map obtained after the merging operation is cascaded with the original output feature map and input into the classification network layer of the classification network, and the network parameters of the classification network are updated using the stochastic gradient descent algorithm.

[0071] In step S202, it is determined whether the current target convolution block is the last convolution block of the classification network. If so, step S203 is executed; otherwise, step S201 is repeated.

[0072] In an embodiment of the present invention, after completing one stage of training, if it is determined that the current target convolution block is not the last convolution block of the classification network, it indicates that the classification network needs to be further trained. At this time, the above-mentioned block patch learning mechanism is applied to the next convolution block to continue training the classification network.

[0073] In step S203, it is determined that the classification network training is completed.

[0074] In an embodiment of the present invention, if the current target convolution block is the last convolution block of the classification network, it indicates that the block patch learning mechanism has been applied to the top layer of the classification network. At this time, the trained classification network can be used to predict the category activation map of the input image.

[0075] The code reference for the classification network training using the progressive implicit fusion training strategy is as follows:

[0076]

[0077] In an embodiment of the present invention, feature maps of different granularities are decomposed into local patches, and the classification network is gradually trained in multiple stages, thereby effectively reusing the local perception ability of the information of the last granularity learned in the previous stage, and allowing the classification network training to focus on learning the feature map of the current granularity. As the block patch learning mechanism is executed, features of different granularities are gradually implicitly integrated into the network, so that the classification network has local granularity perception capabilities with different sensitivities, and enables the classification network to achieve better coverage of the target object area of ​​the image.

[0078] Example 3:

[0079] Figure 3 The implementation process of training a classification network using an explicit fusion training strategy provided in the third embodiment of the present invention is shown. For ease of explanation, only the part related to the embodiment of the present invention is shown, which is detailed as follows:

[0080] Objects in an image may require different trade-offs between local details and global features at different scales and semantic levels. To achieve this goal, this embodiment uses an explicit fusion training strategy to train the classification network. In this embodiment of the present invention, a normal multi-label classification network that does not use a patch learning mechanism in the convolution block can be first trained at the image level, that is, the aforementioned preliminarily trained classification network. Then, the patch learning mechanism is simultaneously applied to different convolution blocks of the classification network, and the corresponding feature maps are concatenated to obtain the final fused feature map. The corresponding convolution layer parameters are updated, and only the parameters of the network layer after the corresponding feature map destruction and subsequent layers are updated.

[0081] Specifically, a patch learning mechanism is used to train a classification network that introduces a multi-branch shared network, wherein the target convolution blocks using the patch learning mechanism in each network branch of the multi-branch shared network are different, and the feature maps of the target convolution blocks are the same and the weights are shared. Figure 3As shown, a multi-branch shared network is introduced into the above-mentioned preliminarily trained classification network. The target convolution blocks using the above-mentioned patch learning mechanism are located in different positions in each branch and are shared. The spliced ​​feature maps obtained by all network branches of the multi-branch shared network and the original output feature maps obtained by the network branches to which the above-mentioned patch learning mechanism is not applied are cascaded and input into the classification network layer.

[0082] In the training process of the classification network that introduces a multi-branch shared network, specifically, a preset scattering operation is performed on the output feature map of the target convolution block, and the multiple first feature blocks obtained after the scattering operation are input into the network layer after the target convolution block to obtain multiple second feature blocks. A preset merging operation is performed on the multiple second feature blocks, and the multiple spliced ​​feature maps obtained after the merging operation are cascaded with the original output feature map and input into the classification network layer of the classification network. The random gradient descent algorithm is used to update the network parameters of the classification network. During the classification network training process, the network parameters of each target convolution block and the subsequent network layer are updated, and the parameters of the network layer before the above-mentioned target convolution block are not updated, that is, Figure 3 The parameters of the network layer before the cross (×) are fixed and not updated. Compared with the progressive implicit fusion training strategy described in Example 2, the explicit fusion training strategy only needs to be trained once. Therefore, the explicit fusion training strategy is more effective than the progressive implicit fusion training strategy, but it will occupy more existing memory.

[0083] In an embodiment of the present invention, a classification network that introduces a multi-branch shared network is trained using a dicing patch learning mechanism, wherein the target convolution blocks using the dicing patch learning mechanism in each network branch of the multi-branch shared network are different, and the weights of the feature maps of the target convolution blocks are shared at the same position, thereby destroying the feature maps at different positions and completing the training of the classification network in one step, so that the classification network can achieve a better coverage effect of the target object area of ​​the image.

[0084] Example 4:

[0085] This embodiment illustrates the methods described in Embodiments 1 to 3 with reference to specific experimental examples.

[0086] This experiment verifies the effectiveness of the above method on the public dataset PASCAL VOC 2012. The evaluation metric used is the commonly used semantic segmentation evaluation metric - Mean Intersection over Union (mIoU). This experiment compares the quality of the CAM map obtained by training the classification network using the methods described in Example 2 and Example 3 with some existing methods. The performance comparison results are shown in Table 1 and Table 2, and are also shown in Table 3. Figure 4A and Figure 4BThe CAM image quality obtained after training the classification network using the methods described in Example 2 and Example 3 is further demonstrated.

[0087] Table 1: CAM image quality evaluation compared with similar methods

[0088]

[0089] As shown in Table 1, the ours(implicit) row in Table 1 indicates that the classification network is trained using the method described in Example 2, and the ours(explicit) row in Table 1 indicates that the classification network is trained using the method described in Example 3. It can be seen from Table 1 that the quality of the CAM image obtained after training the classification network using the methods described in Example 2 and Example 3 (the seed column in Table 1) is higher than that of other methods, and the quality of the final pseudo-label segmentation map obtained by refining the initial pseudo-label segmentation map of the CAM map (the Refinement column in Table 1) is also higher.

[0090] As shown in Table 2, the above-mentioned final pseudo-label segmentation map is used to supervise the training of the DeepLab-v2 semantic segmentation network, and finally evaluated on the PASCAL VOC 2012 validation set (the val column in Table 2) and the test set (the test column in Table 2). The performance of the segmentation network obtained by the method described in the above embodiment is better than that of other methods. In order to further verify the effectiveness of the methods described in Examples 1 to 3.

[0091] Table 2: Performance of the semantic segmentation network supervised by the final pseudo-label segmentation map obtained by the method described in the above embodiment

[0092]

[0093] like Figure 4A As shown, this experimental example compares the accuracy (Precision) and recall (Recall) of CAM image pixel calculation obtained after training the classification network using the methods described in Example 2 and Example 3, and gives the comparison results with other methods. Figure 4A The horizontal axis represents the recall rate (Recall), and the vertical axis represents the precision rate (Precision). The ourExplicit-ResNet50 curve is the precision and recall rate curve of the CAM image obtained after the classification network is trained using the method described in Example 2. The ourExplicit-ResNet50 curve is the precision and recall rate curve of the CAM image obtained after the classification network is trained using the method described in Example 3. Figure 4AIt can be seen that the CAM image quality obtained after training the classification network using the methods described in Example 2 and Example 3 has higher performance. The performance of the CAM image obtained after training the classification network using the methods described in Example 2 and Example 3 under different thresholds is as follows Figure 4B As shown, Figure 4B The implicit PPL curve is the performance of the CAM graph obtained after the classification network is trained using the method described in Example 2 at different thresholds, and the explicit PPL curve is the performance of the CAM graph obtained after the classification network is trained using the method described in Example 3 at different thresholds.

[0094] This experimental example also visualizes the CAM images of some input images in the dataset, such as Figure 4C As shown, Figure 4C The InputImage column is the input image, and the Implicit column and the explicit column are the CAM images obtained after training the classification network using the methods described in Example 2 and Example 3. Figure 4D Visualize some semantic segmentation prediction results in the figure, Figure 4D The Ours row is the prediction result graph of the semantic segmentation network trained using the method described in the first embodiment above.

[0095] From the above experimental data, it can be seen that the CAM images obtained after training the classification network using the progressive implicit fusion training strategy and the explicit fusion training strategy are of higher quality. Furthermore, the performance of the semantic segmentation network trained based on the higher quality CAM images is also improved accordingly.

[0096] Embodiment 5:

[0097] Figure 5 The structure of the semantic segmentation network training device based on patch learning provided by the fifth embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, including:

[0098] A classification network training unit 51 is configured to train the classification network using a preset dicing and patching learning mechanism to obtain a trained classification network, wherein the dicing and patching learning mechanism is configured to divide the output feature map of the target convolution block into a plurality of first feature blocks, and input each first feature block into a network layer following the target convolution block to obtain a plurality of second feature blocks. A concatenated feature map obtained by concatenating the plurality of second feature blocks is fused with the original output feature map and input into the classification network layer. The network parameters of the network layer preceding the target convolution block are not updated.

[0099] A category activation map generation unit is used to predict the category activation map of the training sample using the trained classification network;

[0100] a pseudo-label acquisition unit, configured to acquire a pseudo-label segmentation map of the training sample based on the class activation map; and

[0101] The segmentation network training unit is used to train the semantic segmentation network based on the training samples and the pseudo-label segmentation map to obtain a trained semantic segmentation network.

[0102] Preferably, the scattering operation is used to divide the output feature map of the target convolution block into equal parts and then input the divided parts into the first shared convolution layer to obtain a plurality of first feature blocks;

[0103] The merging operation is used to input the multiple second feature blocks into the second shared convolutional layer to obtain multiple third feature blocks, and the multiple third feature blocks are spliced ​​to obtain a spliced ​​feature map.

[0104] Preferably, the classification network training unit includes:

[0105] An implicit fusion training unit, configured to train the classification network using a patch learning mechanism in a current target convolutional block, where the current target convolutional block is any convolutional block in the classification network;

[0106] The judgment unit is used to judge whether the current target convolution block is the last convolution block of the classification network. If so, it is determined that the classification network training is completed; otherwise, the next convolution block of the current target convolution block is used as the current target convolution block, and the implicit training unit is triggered to execute the step of using the block patch learning mechanism in the current target convolution block to train the classification network.

[0107] Preferably, the classification network training unit further includes:

[0108] A training unit is displayed, which is used to train a classification network that introduces a multi-branch shared network using a dicing patch learning mechanism, wherein the multi-branch shared network includes multiple network branches, and the target convolution blocks using the dicing patch learning mechanism in each network branch are different, and the feature maps of the target convolution blocks have the same position weights shared.

[0109] In the embodiments of the present invention, each unit of the semantic segmentation network training device based on patch learning can be implemented by a corresponding hardware or software unit. Each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, without limiting the present invention. The specific implementation of each unit of the semantic segmentation network training device based on patch learning can be referred to the description of the aforementioned method embodiment and will not be repeated here.

[0110] Example 6:

[0111] Figure 6The structure of the electronic device provided by the sixth embodiment of the present invention is shown. For the convenience of description, only the parts related to the embodiment of the present invention are shown.

[0112] The electronic device 6 of the embodiment of the present invention includes a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned method embodiments are implemented, for example, steps S101 to S104 shown in FIG1 . Alternatively, when the processor 60 executes the computer program 62, the functions of the units in the above-mentioned device embodiments are implemented, for example, Figure 5 The functions of the units 51 to 54 are shown.

[0113] In an embodiment of the present invention, a preset dicing and patching learning mechanism is used to train a classification network to obtain a trained classification network, wherein the dicing and patching learning mechanism is used to perform a preset scattering operation on the output feature map of the target convolution block, and input the multiple first feature blocks obtained after the scattering operation into the network layer after the target convolution block to obtain multiple second feature blocks, and perform a preset merging operation on the multiple second feature blocks. The spliced ​​feature map obtained after the merging operation is cascaded with the original output feature map and then input into the classification network layer of the classification network, and the trained classification network is used to predict the category activation map of the training sample, and the pseudo-label segmentation map of the training sample is obtained based on the category activation map. Based on the training sample and the pseudo-label segmentation map, the semantic segmentation network is trained to obtain a trained semantic segmentation network, thereby improving the quality of the category activation map by applying the dicing and patching learning mechanism in the convolution block, thereby improving the training effect of the segmentation network.

[0114] Embodiment seven:

[0115] In an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps in the above method embodiment are implemented, for example, steps S101 to S104 shown in FIG1 . Alternatively, when the computer program is executed by a processor, the functions of each unit in the above device embodiment are implemented, for example, Figure 5 The functions of the units 51 to 54 are shown.

[0116] In an embodiment of the present invention, a preset dicing and patching learning mechanism is used to train a classification network to obtain a trained classification network, wherein the dicing and patching learning mechanism is used to perform a preset scattering operation on the output feature map of the target convolution block, and input the multiple first feature blocks obtained after the scattering operation into the network layer after the target convolution block to obtain multiple second feature blocks, and perform a preset merging operation on the multiple second feature blocks. The spliced ​​feature map obtained after the merging operation is cascaded with the original output feature map and then input into the classification network layer of the classification network, and the trained classification network is used to predict the category activation map of the training sample, and the pseudo-label segmentation map of the training sample is obtained based on the category activation map. Based on the training sample and the pseudo-label segmentation map, the semantic segmentation network is trained to obtain a trained semantic segmentation network, thereby improving the quality of the category activation map by applying the dicing and patching learning mechanism in the convolution block, thereby improving the training effect of the segmentation network.

[0117] The computer-readable storage medium of the embodiment of the present invention may include any entity, device, or recording medium capable of carrying computer program code, for example, ROM / RAM, magnetic disk, optical disk, flash memory, or other memory.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A semantic segmentation network training method based on patch learning, characterized in that: The method comprises: The classification network is trained using a preset dicing and patching learning mechanism to obtain a trained classification network, wherein the dicing and patching learning mechanism is used to perform a preset scattering operation on the output feature map of the target convolution block describing the global features of the image, the scattering operation is used to divide the output feature map of the target convolution block into equal parts and input the resultant into a first shared convolution layer to obtain a plurality of first feature blocks, the plurality of first feature blocks are input into a network layer following the target convolution block to obtain a plurality of second feature blocks, a preset merging operation is performed on the plurality of second feature blocks in the original order, and the spliced ​​feature map obtained after the merging operation is cascaded with the original output feature map and input into the classification network layer of the classification network; Use the trained classification network to predict the category activation map of the training sample; Obtaining a pseudo-label segmentation map of the training sample based on the class activation map; Based on the training samples and the pseudo-label segmentation map, a semantic segmentation network is trained to obtain a trained semantic segmentation network.

2. The method according to claim 1, wherein The merging operation is used to input the multiple second feature blocks into the second shared convolutional layer to obtain multiple third feature blocks, and to splice the multiple third feature blocks to obtain the spliced ​​feature map.

3. The method according to claim 1, wherein The steps for training a classification network using the preset patch learning mechanism include: Using the patch learning mechanism in a current target convolutional block to train the classification network, wherein the current target convolutional block is any convolutional block in the classification network; Determine whether the current target convolution block is the last convolution block of the classification network. If so, determine that the classification network training is completed; otherwise, use the next convolution block of the current target convolution block as the current target convolution block, and jump to the step of using the block patch learning mechanism in the current target convolution block to train the classification network.

4. The method according to claim 1, wherein The steps of training the classification network using the preset patch learning mechanism also include: The dicing patch learning mechanism is used to train a classification network that introduces a multi-branch shared network, wherein the multi-branch shared network includes multiple network branches, and the target convolution blocks using the dicing patch learning mechanism in each of the network branches are different, and the feature maps of the target convolution blocks have the same position weights shared.

5. A semantic segmentation network training device based on patch learning, characterized in that: The device comprises: a classification network training unit, configured to train the classification network using a preset dicing and patching learning mechanism to obtain a trained classification network, wherein the dicing and patching learning mechanism is configured to perform a preset scattering operation on the output feature map of the target convolution block describing the global features of the image, the scattering operation being configured to equally divide the output feature map of the target convolution block and input the resultant into a first shared convolution layer to obtain a plurality of first feature blocks, input the plurality of first feature blocks into a network layer following the target convolution block to obtain a plurality of second feature blocks, perform a preset merging operation on the plurality of second feature blocks in their original order, concatenate the spliced ​​feature map obtained after the merging operation with the original output feature map, and input the resultant into the classification network layer of the classification network; a class activation map generating unit, configured to use the trained classification network to predict the class activation map of the training sample; a pseudo-label acquisition unit, configured to acquire a pseudo-label segmentation map of the training sample based on the class activation map; and The segmentation network training unit is used to train a semantic segmentation network based on the training samples and the pseudo-label segmentation map to obtain a trained semantic segmentation network.

6. The device according to claim 5, characterized in that The merging operation is used to input the multiple second feature blocks into the second shared convolutional layer to obtain multiple third feature blocks, and to splice the multiple third feature blocks to obtain the spliced ​​feature map.

7. The device according to claim 5, characterized in that The classification network training unit includes: An implicit training unit, configured to use the patch learning mechanism in a current target convolutional block to train the classification network, wherein the current target convolutional block is any convolutional block in the classification network; A judgment unit is used to judge whether the current target convolution block is the last convolution block of the classification network. If so, it is determined that the classification network training is completed; otherwise, the next convolution block of the current target convolution block is used as the current target convolution block, and the implicit training unit is triggered to execute the step of using the block patch learning mechanism in the current target convolution block to train the classification network.

8. The device according to claim 5, wherein The classification network training unit also includes: A training unit is displayed, which is used to use the block patch learning mechanism to train a classification network that introduces a multi-branch shared network, wherein the multi-branch shared network includes multiple network branches, and the target convolution blocks using the block patch learning mechanism in each of the network branches are different, and the feature maps of the target convolution blocks have the same position weight sharing.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images