Block-based bootstrapped normalization method, system and use thereof
By adopting a block-guided normalization method in the deep learning model, dividing the feature map into blocks and combining global statistical information, the domain generalization problem in the field of computer vision is solved, and the robustness and generalization ability of the model in unknown fields are improved.
Patent Information
- Application Number
- CN202310495779.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-05
- Publication Date
- 2025-10-17
- Estimated Expiration
- 2043-05-05
AI Technical Summary
Existing deep learning models have domain generalization problems in the field of computer vision, especially when the distribution of training datasets and test datasets is very different, resulting in performance degradation. Traditional normalization methods such as batch normalization and instance normalization have poor robustness in unknown domains.
A block-guided normalization method is adopted to divide the feature maps into non-overlapping blocks in the spatial dimension and group them in the channel dimension. Combined with global accumulation and statistical information of each batch, instance and channel two-level block normalization is performed to improve the robustness of the model.
It improves the generalization ability of the model in unknown fields, enhances the robustness of model parameters, enables better adaptation to different input distributions, alleviates the overfitting problem, and improves the performance of tasks such as image classification, object detection, and semantic segmentation.
Smart Images
Figure CN116563300B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of computer vision, especially suitable for image classification, target detection, semantic segmentation, image retrieval and other technologies, mainly relates to a block-guided normalization method, system and its application. BACKGROUND
[0002] The field of computer vision has always been one of the important fields of artificial intelligence development. In image classification, target detection, semantic segmentation and image retrieval tasks, deep learning technology has become the most widely used model. However, the application of these models also faces important challenges, one of the most important of which is the domain generalization problem.
[0003] In the field of machine learning, the distribution of training data set and test data set is often different. If the distribution difference between the training data set and the test data set is too large, the model will be difficult to generalize to the test set, resulting in performance degradation. The domain generalization problem is particularly challenging in the field of computer vision, because the features of images between different domains are very different, resulting in performance degradation of deep learning models.
[0004] In order to solve this problem, researchers have proposed methods such as adversarial learning and instance normalization to provide data enhancement and domain adaptation means for deep learning models. However, these methods have their own shortcomings, such as the need for more computing resources and time, high computational complexity, etc.
[0005] In recent years, many researchers have begun to focus on the role of normalization methods in domain generalization. Normalization is a very important part of deep learning, which can standardize the data distribution to a fixed mean and standard deviation to better train the model. However, typical normalization methods such as batch normalization (BN) and instance normalization (IN) do not perform well in unknown domains, and their performance in domain generalization is not satisfactory. SUMMARY
[0006] The present application is aimed at the problem of poor robustness of single-source domain generalization model in the prior art, and provides a block-guided normalization method, system and its application. Along the spatial dimension, the feature map is divided into non-overlapping blocks at the instance level for batch normalization; at the channel level, the feature map is grouped according to the number of blocks in the channel dimension, and random block segmentation is applied to each group. The global cumulative statistical information is combined with the statistical information of each batch, and the instance and channel double-level block normalization is used to solve the problem of serious overfitting of single-source domain, which improves the robustness of single-source domain generalization model in classification, detection and segmentation and other scenarios.
[0007] In order to achieve the above object, the technical scheme adopted by the present application is: based on the block guided normalization method, along the spatial dimension, the feature map is divided into non-overlapping blocks at the instance level for batch normalization; at the channel level, the feature map is grouped according to the number of blocks in the channel dimension, and random block segmentation is applied to each group, and the global cumulative statistical information is combined with the statistical information of each batch.
[0008] As an improvement of the present application, the feature map is divided into non-overlapping blocks for normalization, specifically:
[0009]
[0010] Wherein, PBN is the block-based normalization; f is the feature map, with a dimension of NxCxHxW, N and C represent the number of samples and channels respectively, H and W represent the height and width of the feature map respectively; P is the number of feature blocks; γ and β are learnable affine transformation parameters, μ p And σ p is the mean and standard deviation of all channels of the pth block in the feature map, and the mean and standard deviation of the ith channel are represented by And , and the calculation method is as follows:
[0011]
[0012]
[0013] Wherein, ∈ is a constant for numerical stability.
[0014] As an improvement of the present application, the global cumulative statistical information is combined with the original statistical information, and the final statistical information of each block is:
[0015]
[0016] Wherein, is the global cumulative mean statistical quantity of the ith channel, is the global cumulative standard deviation statistical quantity of the ith channel, and λ is the balance parameter of the local statistical quantity and the global statistical quantity.
[0017] As an improvement of the present application, the blocks of the feature map are normalized respectively at the instance level.
[0018] In order to achieve the above object, the technical scheme adopted by the present application is: based on the block guided normalization method, along the spatial dimension, the feature map is divided into non-overlapping blocks at the instance level for batch normalization; at the channel level, the feature map is grouped according to the number of blocks in the channel dimension, and random block segmentation is applied to each group, and the global cumulative statistical information is combined with the statistical information of each batch.
[0019] S1: for each block guided normalization layer, the picture features extracted by the forward layer of the deep neural network are taken as the input;
[0020] S2: initialize hyperparameter lambda, determine block number set, load global cumulative statistics through pre-training model and
[0021] S3: calculate global statistics mu and sigma in a batch, and update statistics and Wherein, the global statistics mu and sigma refer to the mean and standard deviation of C channels, and the statistics of the i-th channel is expressed as:
[0022]
[0023]
[0024] Wherein, N represents sample, H and W represent height and width of feature map respectively, and ∈ is a constant for numerical stability;
[0025] S4: randomly select the number of blocks P of each channel from the block number set, and group the feature map according to the number of blocks in the channel in the channel dimension;
[0026] S5: sequentially normalize each group at the channel level;
[0027] S6: sequentially perform batch normalization after feature map segmentation, and dynamically update global statistics;
[0028] S7: splice the segmented blocks according to the original order to obtain the normalized feature map as the output.
[0029] As an improvement of the application, the method for updating statistics in step S3 and is specifically:
[0030]
[0031] Wherein, is the global cumulative mean statistics, is the global cumulative standard deviation statistics, μ is the local mean statistics of the current batch, and σ is the local standard deviation statistics of the current batch.
[0032] As another improvement of the application, the hyperparameter lambda takes the value of 0.5, and the feature map is segmented into 4 blocks.
[0033] In order to achieve the above purpose, the technical scheme adopted by the application is: a block guide based normalization method system, comprising a computer program, the computer program is executed by a processor to realize the steps of the above method.
[0034] Compared with the prior art, the present application has the beneficial effects that a block-guided normalization method is provided, by processing features in block units, the statistical quantity of each block can be adjusted more finely, and the global statistical information can also be better used to adjust the gradient of each block, compared with the traditional batch normalization method, the differences between local blocks in the image can be fully explored, so that the information of various different data can be used to optimize the parameters of the batch normalization method.
[0035] The diversified information generated by the method of the present application can also enhance the robustness of other parameters in the model, especially when processing cross-domain shifts, improving the performance of domain generalization; in addition, randomly dividing the pixels of all feature maps (i.e. H x W N-dimensional vectors of each channel) into multiple groups and normalizing each group separately does not preserve the local structure information within each group, resulting in less obvious differences.
[0036] The method is not only suitable for image classification, target detection, semantic segmentation and image retrieval tasks, but also has stronger generalization ability and higher robustness for image data in unknown fields. The method of the present application can bring new breakthroughs to the implementation of cross-domain image classification, detection, segmentation and many other fields, and greatly promote the wide application of machine learning technology in actual production and application. BRIEF DESCRIPTION OF DRAWINGS
[0037] Fig. 1(a) is an image for classification;
[0038] Fig. 1(b) is an image for detection and segmentation;
[0039] Figure 2 Fig. 1(c) is a structure schematic comparison diagram of the method of the present application and the traditional method, wherein
[0040] Figure 2 Fig. 1(a) is an image for classification;
[0041] Figure 2 Fig. 1(b) is an image for detection and segmentation;
[0042] Figure 3 Fig. 1(c) is a structure schematic comparison diagram of the method of the present application and the traditional method, wherein
[0043] Figure 4 Fig. 1(c) is a structure schematic comparison diagram of the method of the present application and the traditional method, wherein
[0044] Fig. 5(a) is an analysis comparison diagram of the hyperparameter lambda of the present application;
[0045] Fig. 5(b) is a comparison diagram of the application layer of the block-guided normalization method (PBN) module of the present application. DETAILED DESCRIPTION
[0046] The present application will be further clarified by the following examples, which should be considered as demonstrating but not limiting the scope of the application.
[0047] Example 1
[0048] In the traditional batch normalization (BN) method, we first define the feature map of a batch as f, with the dimension of NxCxHxW, where N and C represent the number of samples and channels, respectively, and H and W represent the height and width of the feature map, respectively. Generally, BN uses global statistics of the entire batch for normalization to standardize all samples in each iteration. Specifically, this can be represented by the following formula:
[0049]
[0050] where γ and β are learnable affine transformation parameters, and μ and σ are the mean and standard deviation of C channels (i.e., the statistics of batch normalization). The statistics of the i-th channel are represented as:
[0051]
[0052]
[0053] In the above equation, it can be observed that BN applies global statistics of a batch to all samples in each iteration. Therefore, the neural network can learn a fixed pattern, causing the final model to overfit to the source domain in the case of lack of diversity in the training set. In addition, the values of γ and β in equation (1) can not be robust when the test samples do not follow the same distribution as the training samples. Therefore, this patent proposes a novel batch normalization method to solve this problem.
[0054] The block-guided normalization method (PBN) divides the feature map into non-overlapping blocks at the instance level along the spatial dimension for batch normalization; at the channel level, the feature map is grouped according to the number of blocks in the channel dimension, and random block partitioning is applied to each group, combining global cumulative statistics with statistics of each batch, to solve the source domain overfitting problem by exploiting the diversity between local blocks within an image, thereby improving the robustness of the cross-domain model.
[0055] The difference of local block statistics is used for batch normalization within each instance to improve the robustness of the model. This method first divides the feature map of each batch into non-overlapping blocks according to the spatial dimension, and independently normalizes each block. In this way, the scale and shift parameters of the batch normalization layer can be better learned, making the biomimetic parameters more robust. Compared with the traditional global normalization method, the block-guided normalization method can better adapt to different input distributions, improving the generalization ability of the model.
[0056] The block at the instance level is normalized respectively, so that the scale and shift parameters of the batch normalization layer can be better learned, making the biomimetic parameters more robust. In this way, the robustness of the model parameters can be enhanced by utilizing the difference between local blocks in the image, so that the model can better adapt to various input distributions.
[0057] Different block division methods are used to adapt to different channel-level local statistical information differences. In this way, pattern recognition can be promoted, and the batch normalization layer can be prevented from overfitting to the source domain dataset between different channels.
[0058] As shown in FIG. 1(a) and FIG. 1(b), where FIG. 1(a) is derived from an image classification dataset, and FIG. 1(b) is derived from a target detection and semantic segmentation dataset, each image is divided into four blocks of different sizes. The first number in the tuple beside each block image represents the mean value of the block image, and the second number represents the standard deviation of the block image. From the figure, we can clearly see that even in the same image, the mean and variance of different blocks are not the same, which indicates that the styles of different blocks of an image are different. The block-guided normalization method utilizes the difference between non-overlapping blocks in the image to enhance the robustness of the cross-domain model. The difference between the traditional batch normalization method and the block-guided normalization method is shown in Figure 2 (a) and Figure 2 (b). Figure 2 (a) is a structural diagram of the traditional batch normalization (BN) method, Figure 2 (b) is a structural diagram of the block-guided normalization method (PBN) of the present application in three dimensions. As can be seen from the figure, in PBN, the feature map is divided into multiple non-overlapping blocks, and each block is independently normalized in the forward process.
[0059] We assume that the number of blocks is represented by P, and Hp and Wp represent the height and width of the pth block. We express PBN as follows:
[0060]
[0061] where μ p and σ p are the channel mean and standard deviation (i.e., statistical information) of the pth block in the feature map. and respectively represent the mean and standard deviation of the i-th channel, which are calculated as follows:
[0062]
[0063]
[0064] Although using this method can alleviate overfitting to the training set, Hp and Wp in equations (5) and (6) are smaller than H and W in equations (2) and (3) (see Figure 2 (a) and Figure 2 (b)), so μ p and σ p may not be accurate compared to the global statistics of the batch in the training process. In PBN, we further combine the globally accumulated statistics with the original statistics of each block, and the final statistics of each block can be updated as:
[0065]
[0066] In the training phase, we randomly select the number of blocks for each channel from a set S (e.g., {1, 2, 4}). To reduce the computational cost, the feature maps are not randomly divided into blocks independently, which requires C iterations in a batch. Instead, the feature maps are divided into G groups according to the number of blocks in the channel dimension, and random block division is applied to each group. Therefore, only G iterations are required. In the testing phase, the globally accumulated statistics are used to normalize the feature maps, which is consistent with the traditional BN.
[0067] The present application provides a simple and effective method to solve the single-source domain generalization problem in classification, object detection, semantic segmentation and instance retrieval. The PBN module is used to replace the traditional BN module, which effectively alleviates the problem of easy overfitting to the source domain in the single-source domain generalization process.
[0068] Example 2
[0069] Take the image classification single-source domain generalization dataset PACS as an example. The PACS dataset consists of four different domains: Photo, Art, Cartoon and Sketch. It contains 9,991 images of 7 object categories, including Photo (1,670 images), Art (2,048 images), Cartoon (2,344 images) and Sketch (3,929 images).
[0070] Implementation details: For the PACS dataset, we use ResNet-18 as the backbone network. We report the average results of five times. Since our PBN can be integrated into most existing SOTA methods, we directly replace all BNs of the backbone network with our PBN.
[0071] As shown in the following Algorithm 1, the algorithm 1 for the block-guided normalization method mainly includes the following steps:
[0072]
[0073] 1) For each PBN layer, the picture features extracted by the forward layer of the deep neural network are taken as the input.
[0074] 2) Step 3 is the preparation work before training.
[0075] 3) Steps 4-5 update the statistics by calculating the global statistics (mu and sigma) within a batch and
[0076] 4) Steps 6-7 are blocked at the instance level and grouped at the channel level.
[0077] 5) Steps 8-15 are sequentially normalized for each group at the channel level.
[0078] 6) Steps 10-14 are sequentially batch normalized after the feature map segmentation block, and the global statistics are dynamically updated.
[0079] 7) Step 15 is to splice the segmented blocks in the original order to obtain the normalized feature map f as the output.
[0080] In step 3 of Algorithm 1, we determine the method of set S through experiments. We conducted experiments on the necessity of randomness in the channel dimension, using different numbers of block sets and randomly selected numbers, as shown in Figure 4 From the experimental results, it can be observed that the “P124-S2” setting achieves good performance, and increasing the block set size and the randomly selected number slightly improves the performance. Therefore, we selected “P124-S2” as the final classification task setting.
[0081] In step 3 of Algorithm 1, we determine lambda by experiment. In Equation 7, the hyper-parameter lambda is used to balance the influence between the global cumulative statistics and the statistics of each patch. We analyze the influence of different lambda values as shown in Figure 5(a). It can be observed from the figure that the best performance is achieved when lambda is set to 0.5. In addition, the performance is significantly reduced when only using the global cumulative statistics, because this method reduces the diversity of information.
[0082] In step 6 of Algorithm 1, we determine the number of patches P and the size of the patches by experiment. In this experiment, we show the results of using different partitioning schemes and patch numbers on the PACS dataset as shown in Table 1. It is worth noting that in order to observe the influence of different partitioning schemes and patch numbers, we use the same scheme on all feature map channels. For the partitioning schemes of 2 and 4 patches, we ensure that each patch is not too small by randomly selecting a position between 1 / 3 and 2 / 3 of the height (or width). For 9 patches, we select two positions in the height and width dimensions (i.e., 1 / 5 to 2 / 5 and 3 / 5 to 4 / 5). As can be seen from the table, the random partitioning scheme generally outperforms the average partitioning scheme. In addition, we find that randomly partitioning the feature maps into 4 patches (i.e., "P4-random") can achieve the best results.
[0083] Table 1
[0084]
[0085]
[0086] where random means uneven division, equal means equalization division, P2 represents division into two patches, LR represents left-right division, and UD represents up-down division.
[0087] In addition to the patch-based batch normalization method, we also conduct a pixel-level guided normalization method for comparison. In this experiment, we further explore the necessity of using patches in our method. We use a pixel-based scheme for this experiment, which randomly divides all pixels in a batch (i.e., an H x W N-dimensional vector for each channel) into multiple pixel groups. In order to make a fair comparison, the number of pixel groups is the same as the number of divided patches, and the size of each group is generated by the same random scheme as the patch division scheme (i.e., "P4-random" in Table 1). The experimental results are shown in Table 2. As shown in the table, the patch-based guided scheme is necessary and meaningful in our method, while the pixel-based scheme ignores the structural information of the spatial dimension and utilizes the differences between local patches to alleviate the overfitting problem of the training set in cross-domain tasks.
[0088] Table 2
[0089] Method P A C S Avg BN (Baseline) 34.0 58.6 66.4 27.5 46.6 Pixel 34.6 60.4 67.9 24.7 46.9 Patch 36.7 61.4 68.9 29.2 49.1
[0090] We use the traditional ResNet-18 as the backbone on the PACS dataset, and our PBN module replaces all BN layers in the backbone. We conduct comprehensive experiments for this setting. ResNet is composed of five modules, i.e., one convolution module and four residual block modules. In Fig. 5(b), we show the experimental results when using PBN at different positions. In the figure, “L1-5” means that we replace the BN of all five modules. As shown in the figure, when all original BNs are replaced with PBN, our method can achieve the best result on PACS. In the classification task, we use this setting for all experiments.
[0091] Table 3
[0092] Method P A C S Avg ERM 38.0 63.5 69.2 31.4 50.5 ERM + PBN 41.4 67.0 71.5 38.0 54.5 EFDMix 48.0 75.3 77.4 44.2 61.2 EFDMix + PBN 55.3 78.2 75.9 46.4 64.0
[0093] Here, ERM represents the baseline, and ERM+PBN represents the addition of our method to the baseline. EFDMix is the abbreviation of the method in the paper Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization, and EFDMix+PBN represents the addition of the PBN module to the EFDMix method.
[0094] To prove that the PBN module has strong robustness on multiple backbones, we further illustrate this. We use ResNet-50 as the backbone to conduct experiments on the PACS dataset. The experimental results are shown in Table 3. For example, when our PBN is integrated into the EFDMix method, it can still increase by +2.8%.
[0095] To prove that the PBN module has strong robustness on multiple datasets, we further conduct experiments on the cifar10 dataset. The experimental results are shown in Table 4. For example, when our PBN module is inserted into the six methods such as ERM, it can still obtain good improvement. Figure 3
[0096] In summary, the method of the present application first divides the feature map into non-overlapping blocks along the spatial dimension, and then independently normalizes each block to jointly optimize the shared block-guided batch normalization method parameters in each iteration. By taking advantage of the differences between local blocks in the image, the proposed block-guided batch normalization method can effectively enhance the robustness of model parameters. In addition, considering that the statistics of each block may be small and inaccurate compared to the statistics of the global feature map, the globally accumulated statistics are combined with the statistics of each batch to obtain the final statistics for normalizing each block. Since the proposed block-guided batch normalization method can replace the typical batch normalization, it can be integrated into most existing state-of-the-art methods. Extensive experiments and analysis demonstrate the effectiveness of our block-guided batch normalization method in multiple computer vision tasks, including classification, object detection, instance retrieval, and semantic segmentation.
[0097] It should be noted that the above only illustrates the technical idea of the present application, and cannot limit the protection scope of the present application. For ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which fall within the scope of protection of the claims of the present application.
Claims
1. Block-guided normalization method, characterized by : Along the spatial dimension, at the instance level, the feature map is divided into non-overlapping blocks for batch normalization; at the channel level, the feature map is grouped according to the number of blocks in the channel dimension, and random block splitting is applied to each group, combining the globally accumulated statistics with the statistics of each batch; Among them, the feature map is divided into non-overlapping blocks for normalization as follows: Among them, PBN is block-based normalization; f is a feature map with dimensions of N×C×H×W, where N and C represent the number of samples and channels, respectively, and H and W represent the height and width of the feature map, respectively; P is the number of feature blocks; γ and β are learnable affine transformation parameters, μ p and σ p is the mean and standard deviation of all channels of the p-th block in the feature map, and the mean and standard deviation of the i-th channel are and It is calculated as follows: where ∈ is a constant used for numerical stability; Combining the globally accumulated statistics with the original statistics, the final statistics for each block are: in, is the global cumulative mean statistic of the i-th channel, is the global cumulative standard deviation statistic of the i-th channel, and λ is the balance parameter between local statistics and global statistics.
2. The block-guided normalization method according to claim 1, wherein: At the instance level, the blocks divided into feature maps are normalized separately.
3. The block-guided normalization method according to claim 1, wherein: The steps include: S1: For each block-guided normalization layer, the image features extracted by the forward layer of the deep neural network are used as input; S2: Initialize the hyperparameter lambda, determine the number of blocks, and load the global cumulative statistics through the pre-trained model and S3: Calculate the global statistics mu and sigma within a batch and update the statistics and Among them, the global statistics mu and sigma refer to the mean and standard deviation of C channels, and the statistics of the i-th channel are expressed as: Where N represents the sample, H and W represent the height and width of the feature map, respectively, and ∈ is a constant for numerical stability; S4: Randomly select the number of blocks P for each channel from the block number set, and group the feature maps according to the number of blocks in the channel in the channel dimension; S5: normalize each group at the channel level in turn; S6: After the feature map is segmented, batch normalization is performed sequentially and global statistics are dynamically updated; S7: Splice the segmented blocks in the original order to obtain the normalized feature map as output.
4. The block-guided normalization method according to claim 3, wherein: In step S3, the statistical data is updated and The specific method is: in, is the global cumulative mean statistic, is the global cumulative standard deviation statistic, μ is the local mean statistic of the current batch, and σ is the local standard deviation statistic of the current batch.
5. The block-guided normalization method according to claim 3, wherein: The hyperparameter lambda takes a value of 0.5, and the feature map is divided into 4 blocks.
6. A system based on a block-guided normalization method, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.