HRNet-based high-resolution representation method for marking pixels and areas
By connecting high to low convolutional flows in HRNet in parallel and performing multi-scale fusion, the problem of insufficient spatial sensitivity in high-resolution representations is solved, and higher accuracy is achieved in tasks such as semantic segmentation. Characterize learning and representation.
Patent Information
- Application Number
- CN202411082125.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-15
AI Technical Summary
The characterization learned by existing convolutional neural networks in image classification has low resolution characteristics, making it difficult to obtain accurate prediction results on tasks with spatial accuracy sensitive, especially in scenarios where high resolution representation demands such as semantic segmentation. There is a high resolution indicating that the recovery method has insufficient spatial sensitivity.
Using HRNet-based parallel high-resolution representation method, the high-resolution to low-resolution subnets are gradually increased by parallel connection of high-to-low convolutional flows, and multi-scale fusion is carried out to design batch normalization and residual connections. , maintain high resolution representation, and achieve cross-resolution information interaction and semantic fusion.
It realizes learning the representation of strong spatial accuracy in scenarios with high precision requirements such as semantic segmentation, improves the spatial sensitivity and semantic expression ability of high-resolution representations, and provides reliable low-resolution representations in tasks such as object detection and classification. .
Smart Images

Figure CN118967445A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition and segmentation, and in particular to a high-resolution representation method for marking pixels and regions based on HRNet. Background Art
[0002] Convolutional neural networks for image classification were once the standard structure for solving visual recognition problems, but classification networks are not suitable for regional and pixel-level problems because the representations learned by these networks are inherently low-resolution. The huge loss in resolution makes it difficult to achieve accurate prediction results in tasks that are sensitive to spatial accuracy. Therefore, there are two main representation methods: low-resolution representation is mainly used for image classification, and high-resolution representation plays a vital role in many visual problems, such as semantic segmentation, object detection, human pose estimation, etc. High-resolution representation is attracting widespread attention.
[0003] In the prior art, there are generally two methods for calculating high-resolution representations. One is to restore high-resolution representations from low-resolution representations and optional medium-resolution representations output by the network, such as Hourglass. Since it has gone through two processes, High-to-Low (generating low-resolution high-level feature representations) and Low-to-High (restoring high resolution), and these two processes are serial, the final high-resolution representation mainly comes from two parts: the first is the original high-resolution representation, but since it has only undergone a small number of convolution operations, it can only provide low-level semantic expressions; the second is the high-resolution representation obtained by upsampling the low-resolution representation. Although it has good semantic expression capabilities, upsampling itself cannot completely make up for the loss of spatial resolution. Therefore, the spatial sensitivity of the high-resolution representation outputted in the end is not high, and is largely limited by the resolution corresponding to the representation with strong semantic expression. Summary of the invention
[0004] The purpose of the present invention is to provide a high-resolution representation method for marking pixels and regions based on HRNet, and to learn a representation with strong spatial accuracy. HRNet starts from scratch and maintains high resolution from beginning to end, which embodies a representation with strong spatial resolution. It combines the representations of all high-resolution to low-resolution parallel streams, and can be applied to scenarios with higher accuracy requirements such as semantic segmentation; so as to solve the problems raised in the above background technology.
[0005] To achieve the above object, the present invention provides the following technical solution: a high-resolution representation method for marking pixels and regions based on HRNet, the method comprising the following steps:
[0006] HRNet connects all high-to-low convolution flows in parallel;
[0007] Starting from a high-resolution subnet as the first stage, gradually increase the high-resolution to low-resolution subnets to form more stages, and connect the multi-resolution subnets in parallel;
[0008] Repeated multi-scale fusion is performed by repeatedly exchanging information across parallel subnetworks at multiple resolutions;
[0009] Output the result on the high-resolution convolutional stream representation of the network output;
[0010] Design batch normalization and residual connections.
[0011] Preferably, HRNet connects all high-to-low convolutional streams in parallel to maintain high-resolution representations throughout the process, and generates reliable high-resolution representations by repeatedly fusing representations of multi-resolution streams.
[0012] Preferably, batch normalization performs an operation similar to image whitening on the input of each layer, so that the data of each layer is controlled within a stable distribution.
[0013] Preferably, the normalization step specifically includes:
[0014] The first step is to find the mean μ of each training batch of data B :
[0015]
[0016] The second step is to find the variance of each training batch of data
[0017]
[0018] The third step is to use the obtained mean and variance to normalize the training data of this batch to obtain a (0, 1) normal distribution. Where ε is a small positive number used to avoid division by 0;
[0019]
[0020] Step 4: Scale transformation and offset: Multiply by γ to adjust the value, and add β to increase the offset to get y i , γ is the scale factor, β is the translation factor; two new parameters are introduced: γ, β, which are learned by the network itself during training.
[0021]
[0022] Compared with the prior art, the present invention has the following beneficial effects:
[0023] The high-resolution representation method for marking pixels and regions based on HRNet proposed in the present invention connects high-resolution to low-resolution subnetworks in parallel, rather than in series like most existing solutions, which enables the network to maintain high resolution throughout the entire processing process, rather than needing to restore high resolution through a high-to-low and then low-to-high process like previous methods, which makes the network potentially more accurate in space;
[0024] HRNet can repeatedly perform multi-scale fusion. It retains the original extracted high-resolution feature map and generates a low-resolution feature map stream at an appropriate location. The semantic information between each resolution is relatively independent. The high-resolution representation can be improved with the help of low-resolution representations of the same depth and similar level. Semantic fusion is achieved through feature fusion of cross-resolution information interaction, which adds rich semantics on the basis of the original spatial information accuracy. Moreover, the network can also use high-resolution representation to improve low-resolution representation, resulting in high-resolution representation can also enrich the output results, which enables it to provide reliable low-resolution representation in applications such as target detection and classification that do not require high spatial accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is the structural model diagram of the HRNet network of the present invention;
[0026] Figure 2 This is a model diagram of stage 3 and stage 4 of the present invention. DETAILED DESCRIPTION
[0027] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] See also Figure 1 to Figure 2 The present invention provides a technical solution: a high-resolution representation method for marking pixels and regions based on HRNet, the method comprising the following steps:
[0029] HRNet connects all high-to-low convolution flows in parallel, and maintains high-resolution representation throughout the process. It generates reliable high-resolution representation by repeatedly fusing the representations of multi-resolution flows, and has strong position sensitivity. The network starts with a high-resolution subnet as the first stage, gradually adds high-resolution to low-resolution subnets, forms more stages, and connects multi-resolution subnets in parallel. Throughout the process, repeated multi-scale fusion is performed by repeatedly exchanging information across multi-resolution parallel subnets. And the result is output on the high-resolution convolution flow representation output by the network. Maintaining high-resolution representation is generally achieved by obtaining feature map flows of different resolutions through parallel convolution flows. In the main process of the entire framework, the high-resolution feature map remains unchanged, so that better and more accurate pixel location information can be obtained. Although accurate pixel location information is retained, the semantic information in each pixel is indeed limited, which directly leads to limited semantics of the entire image aggregation, resulting in limited performance in segmentation tasks. For this, batch normalization and residual connections need to be designed.
[0030] Batch normalization performs an operation similar to image whitening on the input of each layer, keeping the data of each layer within a stable distribution.
[0031] The normalization step is mainly divided into 4 steps. The first step is to find the mean μ of each training batch data. B :
[0032]
[0033] The second step is to find the variance of each training batch of data
[0034]
[0035] The third step is to use the obtained mean and variance to normalize the training data of this batch to obtain a (0, 1) normal distribution. Here ε is a small positive number used to avoid division by 0.
[0036]
[0037] Step 4: Scale transformation and offset: Multiply by γ to adjust the value, and add β to increase the offset to get y i , where γ is the scale factor and β is the translation factor. This step is the essence of BN. Basically, it will be restricted to the normal distribution, which reduces the network's expressive power. To solve this problem, two new parameters can be introduced: γ and β. These two parameters are learned by the network itself during training.
[0038]
[0039] Residual connections are designed to alleviate the vanishing problem and make forward and backward information propagation smoother. If a nonlinear change function is used to describe the input and output of a network, let the input be X and the output be F(x), where F usually includes operations such as convolution and activation. When an input is added to the output of the function, although G(x) can still be used to describe the relationship between the input and output, this G(x) can be clearly split into the linear superposition of F(x) and X. The idea of residual connections is to express the output y as the linear superposition of the input and a nonlinear transformation of the input:
[0040] The main network structure of y=H(x,WH)+XHRNet is as follows Figure 1 As shown in the figure, repeated cross-resolution fusion is a major feature of HRNet. The purpose of fusion is to exchange information across multi-resolution representations, and it will be repeated several times throughout the network structure. Stem represents the processing of the input image, converting the image into a feature map that can be input into the main network. It consists of two 3×3 convolutions with a stride of 2, which can reduce the feature map resolution to 1 / 4 of the original image and then input into the main network; head determines the output structure of the HRNetV1 network; depth can be understood as the model depth, which refers to the number of layers contained in the network structure; scale is used to describe the size, which means that each pixel of the feature map in this row of data stream is equivalent to several pixels of the feature map of the stem input main network, such as 1 in the first row, 2 in the second row, 4 in the third row, and 2 in the nth row. n-1 The horizontal line segments represent convolution operations; the line segments with arrows pointing downwards represent downsampling operations, and the line segments with arrows pointing upwards represent upsampling operations; the boxes represent feature maps, and the size of the feature map is proportional to the size of the box area.
[0041] Assume that there are two stages with different scales for cross-resolution fusion transmission. The first stage has 3 resolution sub-streams and the second stage has 4 resolution sub-streams. The two stages are named stage 3 and stage 4 . Let H mn Represents input and output. The subscript m means that the input comes from the stage m , the subscript n means that the input comes from stage m The nth data stream, then the input from top to bottom can be defined as: H 31 ,H 32 ,H 33, then the output can be defined as: H 41 ,H 42 ,H 43 ,H 44 , where H 44 For stage 4 New resolution branch obtained by branch expansion.
[0042] Want to get stage in HRNet 4 The branch needs to be made by using H 31 ,H 32 ,H 33 Let's generate a stage 4 H 41 ,H 42 ,H 43 ,H 44 The aggregation method is to directly add after size matching and channel matching:
[0043] H 41 =f 1 (H 31 )+f 3 (H 33 )+f 3 (H 33 )
[0044] The corresponding feature transformation function before and after fusion (f n ) actually satisfies: For H 41 :
[0045] f 1 (H 31 )=H 31
[0046] Other f 2 ,f 3 It is through upsampling, restoring the feature map size and channel transformation to make it consistent with f 1 (H 31 ) to ensure successful fusion. Similarly, if the fusion H 42 :
[0047] H 42 =f 1 (H 31 )+f 3 (H 33 )+f 3 (H 33 )
[0048] f 2 (H 32 )=H 32
[0049] And f 1 To achieve downsampling, f 3 To achieve upsampling, the rest is still channel matching, matching is f 2 (H 32 ). And so on, all outputs are fused and calculated. m-1 To the stage m When performing cross-resolution information interaction, the newly fused resolution branch results.
[0050] In order to make HRNet more widely used in visual technology, all quadruple resolution representations can be used and the connection between multi-resolution fusion and regular convolution can be established. The main body of this network consists of four stages: Figure 1 It can be seen as a combination of stage1 and stage2. The structure of stage3 and stage4 is as follows Figure 2 As shown. There are four parallel convolution streams with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 respectively. The first stage contains 4 residual units, each of which consists of a bottleneck layer with 64 channels, followed by a 3×3 convolution to change the number of channels of the feature map to C. The second, third, and fourth stages contain 1, 4, and 3 modular blocks respectively. In the multi-resolution parallel convolution of the modular block, each branch contains 4 residual units. Each unit contains two 3×3 convolutions for each resolution, each of which is followed by batch normalization and a nonlinear ReLU activation function. The number of channels for the four resolutions is C, 2C, 4C, and 8C respectively. The head part of the output varies depending on the application of the network. The specific difference is reflected in the diversity of the output. If the output is only the representation from the high-resolution stream, this means that only the results of the high-resolution output channel are utilized, and the output results of other low-resolution channels are lost; HRNet can also adjust the low-resolution representation to high resolution through bilinear upsampling without changing the number of channels connected to the high-resolution representation, and connect the representations of four resolutions, and then mix the four representations through 1×1 convolution; or construct a multi-level resolution representation by performing multi-level downsampling on the high-resolution representation output. Although it does not perform multiple cross-resolution feature fusions, but only performs downsampling once, they are actually equivalent, and feature fusion is performed by splicing along the channel direction.
[0051] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A high-resolution representation method for marking pixels and regions based on HRNet, characterized by: The method comprises the following steps: HRNet connects all high-to-low convolution flows in parallel; Starting from a high-resolution subnet as the first stage, gradually increase the high-resolution to low-resolution subnets to form more stages, and connect the multi-resolution subnets in parallel; Repeated multi-scale fusion is performed by repeatedly exchanging information across parallel subnetworks at multiple resolutions; Output the result on the high-resolution convolutional stream representation of the network output; Design batch normalization and residual connections.
2. A high-resolution representation method for marking pixels and regions based on HRNet according to claim 1, characterized in that: HRNet connects all high-to-low convolutional streams in parallel to maintain high-resolution representations throughout the process and generates reliable high-resolution representations by repeatedly fusing representations of multi-resolution streams.
3. The high-resolution representation method for marking pixels and regions based on HRNet according to claim 1, characterized in that: Batch normalization performs an operation similar to image whitening on the input of each layer, keeping the data of each layer within a stable distribution.
4. The high-resolution representation method for marking pixels and regions based on HRNet according to claim 1, characterized in that: The normalization steps specifically include: The first step is to find the mean μ of each training batch of data B : The second step is to find the variance of each training batch of data The third step is to use the obtained mean and variance to normalize the training data of this batch to obtain a (0, 1) normal distribution. Where ε is a small positive number used to avoid division by 0; Step 4: Scale transformation and offset: Multiply by γ to adjust the value, and add β to increase the offset to get y i , γ is the scale factor, β is the translation factor; two new parameters are introduced: γ, β, which are learned by the network itself during training.
