Cross-channel pyramid pooling method, system, convolutional neural network and processing method
By using cross-channel pyramid pooling to extract pixel semantic features at different channel scales at the same pixel location in a convolutional neural network, the problem of not considering the differences in channel dimension contextual information in existing spatial pyramid pooling methods is solved, thus improving the performance of image classification and object detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-30
- Publication Date
- 2026-03-17
AI Technical Summary
Existing spatial pyramid pooling methods fail to effectively consider the differences in contextual information between pixels at the same pixel location in the channel dimension in convolutional neural networks, resulting in the loss of contextual information between different sub-regions of the feature map.
A cross-channel pyramid pooling method is proposed. By setting multiple channel scales and performing cross-channel pooling operations under multiple channel sets, the semantic features of pixels at different channel scales at the same pixel position are extracted, taking into account the distribution differences between channels.
It improves the performance of convolutional neural networks, especially in image classification, object detection and segmentation tasks, by enhancing the prominence of important pixel locations and optimizing prediction results.
Smart Images
Figure CN115984666B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer image processing technology, and in particular to a cross-channel pyramid pooling method, system, convolutional neural network, and processing method. Background Technology
[0002] In computer vision and medical image analysis, classification is a fundamental and crucial task. For example, in cataract classification, convolutional neural networks (CNNs) are typically used to automatically extract representational information from ophthalmic images to output classification results. In CNNs, such as the classic Residual Neural Network (ResNet) architecture, Global Average Pooling (GAP) is commonly used to compress feature maps. However, GAP typically compresses the feature map into a single global feature representation, which can easily lead to the loss of contextual information between different sub-regions of the feature map and fails to consider the differences in contextual information between pixels at the same pixel location along the channel dimension.
[0003] To reduce the loss of contextual information between different sub-regions in a feature map, Spatial Pyramid Pooling (SPP) has been proposed by researchers. Its function is to extract feature representations of regions at different scales from the feature map, further reducing the loss of contextual information between different sub-regions. For example, three feature maps of different scales can be extracted along the spatial dimension from an H×W feature map and pooled using mean pooling, such as 1x1, 2x2, and 4x4. However, existing Spatial Pyramid Pooling methods only consider the importance of contextual information between regions in the spatial dimension, without considering the differences in contextual information between pixels at the same pixel location in the channel dimension. Summary of the Invention
[0004] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a cross-channel pyramid pooling method, which solves the problem that existing spatial pyramid pooling methods lack consideration for the differences in contextual information between pixels at the same pixel location along the channel dimension.
[0005] The present invention also provides a cross-channel pyramid pooling system, a convolutional neural network, a convolutional neural network processing method, and a computer-readable storage medium.
[0006] The cross-channel pyramid pooling method according to a first aspect of the present invention includes the following steps:
[0007] Multiple different channel scales are set, and each of the different channel scales corresponds to a different cross-channel pooling operation;
[0008] Obtain multiple channels at each of the aforementioned channel scales to obtain multiple channel sets;
[0009] A multi-layer feature map is obtained by processing the input image through a convolutional layer.
[0010] The multi-layer feature map is subjected to cross-channel pooling operations under multiple channel sets to obtain pooling results.
[0011] The cross-channel pyramid pooling method according to embodiments of the present invention has at least the following beneficial effects:
[0012] By improving the spatial pyramid pooling method, specifically by incorporating the principle of cross-channel pooling, the cross-channel pyramid pooling method of this invention is obtained. This method extracts pixel semantic features at different channel scales for the same pixel location along the channel dimension, considering the distribution differences of pixel semantic features between channels. This helps to highlight important pixel locations and overcomes the shortcomings of existing spatial pyramid pooling methods. Therefore, the cross-channel pyramid pooling method of this invention can be applied to neural network architectures, such as convolutional neural networks, Transformers, and multilayer perceptrons (MLPs), to improve performance. It can be used for two-dimensional and three-dimensional image classification, image segmentation, object detection and localization, and other computer vision tasks. Furthermore, by embedding it into a convolutional neural network framework as a main module, it can be deployed to servers, workstations, supercomputing centers, cloud computing centers, personal computers, mobile phones, edge devices, and medical devices.
[0013] According to some embodiments of the present invention, the channel scale corresponding to the Nth channel set is 2. (N-1) N is a positive integer; the step of performing cross-channel pooling operations on the multi-layer feature map under multiple channel sets includes the following steps:
[0014] The multi-layer feature map is divided into M pixel regions, and the M pixel regions together form the multi-layer feature map, where M is a positive integer.
[0015] Perform cross-channel pooling on the first pixel region across N channel sets to obtain 2 N -1 number of pixel semantic features;
[0016] Continue performing cross-channel pooling operations on the remaining M-1 pixel regions under the N channel sets to obtain M(2 N -1) Number of pixel semantic features;
[0017] For M(2) N -1) A number of pixel semantic features are merged to obtain pixel semantic feature data, which represents the cross-channel pyramid pooling result.
[0018] According to some embodiments of the present invention, the cross-channel pooling operation employs mean cross-channel pooling.
[0019] According to some embodiments of the present invention, the mean cross-channel pooling operation is constrained by the following mathematical model:
[0020]
[0021] K = W / 2 (N-1) ,
[0022] Where W is the number of channels in each channel set; 2 (N-1) The channel scale corresponding to the Nth channel set is represented by x(k, i, j); x(k, i, j) represents the kth pixel region, where k is a positive integer and not greater than K; μ (i，j) Represents the semantic features of pixels.
[0023] According to some embodiments of the present invention, the cross-channel pooling operation employs a maximum cross-channel pooling operation.
[0024] According to some embodiments of the present invention, the maximum cross-channel pooling operation is constrained by the following mathematical model:
[0025] μ (i，j) =max{x(k,i,j)|k∈[1,K],K∈N};
[0026] K = W / 2 (N-1) ;
[0027] Where W is the number of channels in each channel set; 2 (N-1) The channel scale corresponding to the Nth channel set is represented by x(k, i, j); x(k, i, j) represents the kth pixel region, where k is a positive integer and not greater than K; μ (i，j) Represents the semantic features of pixels.
[0028] A cross-channel pyramid pooling system according to a second aspect of the present invention includes:
[0029] The channel scale setting unit is used to set multiple channel scales, each of which represents a different cross-channel pooling operation.
[0030] The channel set acquisition unit is used to acquire multiple channels at each of the channel scales to obtain multiple channel sets;
[0031] The feature map acquisition unit is used to acquire multi-layer feature maps, which are obtained by processing the input image through a convolutional layer.
[0032] The cross-channel pooling unit is used to perform cross-channel pooling operations on the multi-layer feature map under multiple channel sets to obtain the pooling result.
[0033] The cross-channel pyramid pooling system according to embodiments of the present invention has at least the following beneficial effects:
[0034] By improving the spatial pyramid pooling method, specifically incorporating the principle of cross-channel pooling, a cross-channel pyramid pooling system is obtained according to this invention. This system extracts pixel semantic features at different channel scales for the same pixel location along the channel dimension, considering the distribution differences of pixel semantic features between channels. This helps to highlight important pixel locations and overcomes the shortcomings of existing spatial pyramid pooling methods. Therefore, the cross-channel pyramid pooling system of this invention can be applied to neural network architectures such as convolutional neural networks, Transformers, and multilayer perceptrons (MLPs) to improve performance. It can be used for two-dimensional and three-dimensional image classification, image segmentation, object detection and localization, and other computer vision tasks. Furthermore, by embedding it into a convolutional neural network framework as a main module, it can be deployed to servers, workstations, supercomputing centers, cloud computing centers, personal computers, mobile phones, edge devices, and medical devices.
[0035] A convolutional neural network according to a third aspect of the present invention includes an input layer, a convolutional layer, a pooling layer, a cross-channel pyramid pooling layer, a fully connected layer, and an output layer, wherein the cross-channel pyramid pooling layer is used to perform the cross-channel pyramid pooling method as described in any of the first aspects of the present invention.
[0036] The convolutional neural network according to embodiments of the present invention has at least the following beneficial effects:
[0037] By utilizing the convolutional neural network of this invention, optimized output results can be obtained. Specifically, the optimization is achieved through the use of a cross-channel pyramid pooling layer. This layer extracts pixel semantic features at different channel scales for the same pixel location along the channel dimension, taking into account the distribution differences of pixel semantic features across channels. This approach helps to highlight important pixel locations and overcomes the shortcomings of existing spatial pyramid pooling methods. Therefore, the convolutional neural network of this invention can be deployed in practical applications such as image classification, object detection, and segmentation, and can achieve better prediction results.
[0038] According to a fourth aspect of the present invention, a convolutional neural network processing method applied to a convolutional neural network as described in a third aspect embodiment includes the following steps:
[0039] The input image data is processed using the input layer, the convolutional layer, and the pooling layer to obtain high-level feature data, which represents the high-level features extracted from the image.
[0040] The high-level feature data is processed using the cross-channel pyramid pooling layer to obtain pixel semantic feature data, which represents the cross-channel pyramid pooling result.
[0041] The pixel semantic feature data is processed using the fully connected layer and the output layer to obtain the prediction result of the convolutional neural network.
[0042] The convolutional neural network processing method according to embodiments of the present invention has at least the following beneficial effects:
[0043] By applying the convolutional neural network processing method of this invention to the convolutional neural network of this embodiment, optimized output results can be obtained. Specifically, the optimization effect is reflected in the use of a cross-channel pyramid pooling layer. By extracting pixel semantic features at different channel scales for the same pixel location along the channel dimension, the distribution differences of pixel semantic features between channels are considered, which helps to highlight important pixel locations and overcomes the shortcomings of existing spatial pyramid pooling methods. Therefore, by applying the convolutional neural network processing method of this invention to practical applications such as image classification, object detection, and segmentation, better prediction results can be obtained.
[0044] According to a fifth aspect of the present invention, a computer-readable storage medium stores computer-executable instructions for causing a computer to perform a cross-channel pyramid pooling method as described in a first aspect of the present invention or a convolutional neural network processing method as described in a fourth aspect of the present invention.
[0045] It is understood that the beneficial effects of the fifth aspect compared with the related technologies are the same as the beneficial effects of the first or fourth aspect compared with the related technologies. Please refer to the relevant descriptions in the first or fourth aspects above, which will not be repeated here.
[0046] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description
[0047] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:
[0048] Figure 1 This is a schematic diagram of a spatial pyramid pooling method according to an embodiment of the present invention;
[0049] Figure 2This is a flowchart of a cross-channel pyramid pooling method according to an embodiment of the present invention;
[0050] Figure 3 This is a schematic diagram of a cross-channel pyramid pooling method according to an embodiment of the present invention;
[0051] Figure 4 This is a schematic diagram of a convolutional neural network according to an embodiment of the present invention;
[0052] Figure 5 This is a flowchart of a convolutional neural network processing method according to an embodiment of the present invention. Detailed Implementation
[0053] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals characterize the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0054] In the description of this invention, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated.
[0055] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.
[0056] In the description of this invention, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.
[0057] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are some embodiments of the present invention, not all embodiments.
[0058] It should be noted that, in order to better understand the improvement of the cross-channel pyramid pooling method in the embodiments of the present invention, the spatial pyramid pooling method will be introduced first.
[0059] In convolutional neural networks like R-CNN, a fixed-size image, such as 224×224, is required as the input image. Therefore, a warp operation is performed on all images before input to ensure a consistent input image size. However, this process may reduce the recognition accuracy of images or sub-images of arbitrary sizes and scales. To resolve this contradiction, Spatial Pyramid Pooling (SPP) has been proposed by researchers.
[0060] The idea behind Spatial Pyramid Pooling originates from the Spatial Pyramid Model. It transforms a single pooling operation into multiple pooling operations of varying sizes, applying pooling windows of different sizes to the convolutional features of the preceding layers. In other words, a Spatial Pyramid Pooling layer performs multiple pooling operations on each image in the multi-layered feature maps output from the previous convolutional layer and outputs the results to a fully connected layer. Each pooling operation can be viewed as a layer of a spatial pyramid.
[0061] Specifically, refer to Figure 1 , Figure 1 This is a schematic diagram of the spatial pyramid pooling method. It is understandable that... Figure 1 The algorithm performs a three-layer spatial pyramid pooling operation on the multi-layer feature maps, with three pooling windows of sizes 4×4, 2×2, and 1×1. Assuming the last convolutional layer in the convolutional neural network is conv5, using 256 filters, the generated multi-layer feature maps are of size (W, H, 256), with each layer's feature map size being (W, H). After being input into the spatial pyramid pooling layer, the feature maps undergo max pooling or mean pooling through three pooling windows of different sizes (4×4, 2×2, and 1×1), resulting in feature vectors of levels 16, 4, and 1, respectively. Since there are 256 feature maps in total, the pooling results of each layer are combined to obtain feature vectors of levels 16×256-d, 4×256-d, and 1×256-d. Finally, all feature vectors are concatenated to obtain the final spatial pyramid pooling result.
[0062] See Figure 2 The diagram shows a flowchart of a cross-channel pyramid pooling method provided by an embodiment of the present invention. The method includes the following steps:
[0063] Set up multiple different channel scales, each of which corresponds to a different cross-channel pooling operation;
[0064] Obtain multiple channels at each channel scale to obtain multiple channel sets;
[0065] Obtain multi-layer feature maps, which are obtained by processing the input image through convolutional layers.
[0066] Cross-channel pooling operations are performed on the multi-layer feature map under multiple channel sets to obtain the pooling results.
[0067] Specifically, such as Figure 2 As shown, it can be understood that setting multiple channel scales is equivalent to setting different sizes of the pooling window in the spatial pyramid pooling method; furthermore, obtaining multiple channels at each channel scale to obtain multiple channel sets is equivalent to obtaining pooling windows of different sizes in the spatial pyramid pooling method; and the purpose of obtaining multi-layer feature maps is to obtain input image data for cross-channel pyramid pooling.
[0068] Furthermore, cross-channel pooling is performed on multi-layer feature maps across multiple channel sets. This can be understood as follows: Conventional pooling operations, such as spatial pyramid pooling, pool a single-layer feature map, meaning pooling all pixel data within a single image region. Cross-channel pooling, on the other hand, pools multiple consecutive pixels at the same coordinate position across multiple feature maps. This means that multiple channels in each channel set correspond to multiple layers of the feature map. Therefore, pooling across different channel sets yields corresponding "feature vectors," referred to here as pixel semantic features. Combining several pixel semantic features results in pixel semantic feature data, which serves as the final pooling result. Specifically, refer to the reference... Figure 1 and Figure 3 This allows us to understand the improved features of the cross-channel pyramid pooling method in the embodiments of the present invention.
[0069] In this embodiment, the spatial pyramid pooling method is improved by incorporating the principle of cross-channel pooling, thereby obtaining the cross-channel pyramid pooling method of this invention. This cross-channel pyramid pooling method extracts pixel semantic features at different channel scales for the same pixel location along the channel dimension, considering the distribution differences of pixel semantic features between channels. This helps to highlight important pixel locations and overcomes the shortcomings of existing spatial pyramid pooling methods. Therefore, the cross-channel pyramid pooling method of this invention can be applied to neural network architectures, such as convolutional neural networks, Transformers, and multilayer perceptrons (MLPs), to improve performance. Furthermore, it can be used for two-dimensional and three-dimensional image classification, image segmentation, object detection and localization, and other computer vision tasks. Simultaneously, by embedding it into a convolutional neural network framework as a main module, it can be deployed to servers, workstations, supercomputing centers, cloud computing centers, personal computers, mobile phones, edge devices, and medical devices.
[0070] In some embodiments, the channel scale corresponding to the Nth channel set is 2. (N-1) N is a positive integer; cross-channel pooling operations are performed on the multi-layer feature maps under multiple channel sets, including the following steps:
[0071] The multi-layer feature map is divided into M pixel regions, and the M pixel regions together form the multi-layer feature map, where M is a positive integer;
[0072] Perform cross-channel pooling on the first pixel region across N channel sets to obtain 2 N -1 number of pixel semantic features;
[0073] Continue performing cross-channel pooling operations on the remaining M-1 pixel regions under N channel sets to obtain M(2 N -1) Number of pixel semantic features;
[0074] For M(2) N -1) The number of pixel semantic features are merged to obtain pixel semantic feature data, which represents the cross-channel pyramid pooling result.
[0075] Specifically, refer to Figure 3 , Figure 3 This is a schematic diagram of the cross-channel pyramid pooling method according to an embodiment of the present invention. It can be understood that... Figure 3 Taking three channel sets as an example, according to the definition in this embodiment, the channel scales corresponding to the three channel sets are 1, 2, and 4, respectively. Further, Figure 3 The diagram above uses a pixel region x(i, j) as an example to illustrate pooling operations using the three channel sets mentioned above. A pixel region represents multiple consecutive pixels at the same coordinate position on a multi-layer feature map. After pooling operations with channel scales of 1, 2, and 4, this pixel region yields 1, 2, and 4 pixel semantic features respectively. That is, for the Nth channel set, pooling yields 2... (N-1) The number of pixel semantic features. Therefore, after cross-channel pyramid pooling, the pixel region can obtain 1+2+4=7 pixel semantic features. Mathematically, this can be expressed as: summing a geometric series with a common ratio of 2, that is, any pixel region can obtain 2N-1 pixel semantic features after pyramid pooling.
[0076] Continue to refer to Figure 3 As can be understood from the diagram below, for a multi-layer feature map, all the pixel regions it constitutes can ultimately yield M×7 pixel semantic features, i.e., pixel semantic feature data. Specifically, the pixel semantic feature data is represented by the pixel semantic feature set, whose mathematical model is Y∈R. 7×H×W Where M = H × W. It is understood that this embodiment... Figure 3 Taking only three channel sets as an example, in other embodiments, based on the channel scale definition of this embodiment, a larger number of channel sets can be set to perform cross-channel pyramid pooling, and correspondingly, M(2) can be obtained. N -1) number of pixel semantic features.
[0077] In some embodiments, the cross-channel pooling operation employs the mean cross-channel pooling operation.
[0078] Specifically, it is understandable that, according to relevant theories, feature extraction errors mainly stem from two aspects, one of which is the increased variance of the estimated value due to the limited neighborhood size. To address this feature extraction error, mean pooling can be employed to retain more of the image's background information. It is also understandable that mean pooling across channels is fundamentally similar to mean pooling, and therefore can achieve similar technical results.
[0079] In some embodiments, the mean cross-channel pooling operation is constrained by the following mathematical model:
[0080]
[0081] K = W / 2 (N-1) ,
[0082] Where W is the number of channels in each channel set; 2 (N-1) The channel scale corresponding to the Nth channel set is represented by x(k, i, j); x(k, i, j) represents the kth pixel region, where k is a positive integer and not greater than K; μ (i，j) Represents the semantic features of pixels.
[0083] Specifically, refer to Figure 3 Taking a channel set with a channel scale of 4 as an example, assuming the channel set has 20 channels (i.e., 20 channels), these 20 channels are divided into 4 channel subsets, each containing K=5 channels. Simultaneously, the multi-layer feature map corresponds to 20 layers. Every 5 consecutive layers will sequentially pass through 4 channel subsets for mean-based cross-channel pooling. For example, x(1, i, j), x(2, i, j), x(3, i, j), x(4, i, j), x(5, i, j) undergo mean-based cross-channel pooling through the 5 channels of the first channel subset, essentially performing mean calculation to output a pixel semantic feature μ. (i，j) Therefore, for a channel set with a channel scale of 4, we can ultimately obtain 4 pixel semantic features.
[0084] In some embodiments, cross-channel pooling operations employ maximum cross-channel pooling operations.
[0085] Specifically, it's understandable that, according to relevant theories, feature extraction errors mainly stem from two aspects, one of which is the shift in the estimated mean caused by convolutional layer parameter errors. To address feature extraction errors in this case, max pooling can be employed to preserve more texture information. It's also understandable that max pooling across channels and max pooling are fundamentally similar in principle, thus achieving similar technical results.
[0086] In some embodiments, the maximum cross-channel pooling operation is constrained by the following mathematical model:
[0087] μ (i，j) =max{x(k,i,j)|k∈[1,K],K∈N};
[0088] K = W / 2 (N-1) ;
[0089] Where W is the number of channels in each channel set; 2 (N-1) The channel scale corresponding to the Nth channel set is represented by x(k, i, j); x(k, i, j) represents the kth pixel region, where k is a positive integer and not greater than K; μ (i，j) Represents the semantic features of pixels.
[0090] Specifically, refer to Figure 3 Taking a channel set with a channel scale of 4 as an example, assuming the channel set has 20 channels (i.e., 20 channels), these 20 channels are divided into 4 channel subsets, each containing K=5 channels. Simultaneously, the multi-layer feature map corresponds to 20 layers. Every 5 consecutive layers will sequentially pass through 4 channel subsets for maximum cross-channel pooling. For example, x(1, i, j), x(2, i, j), x(3, i, j), x(4, i, j), x(5, i, j) undergo maximum cross-channel pooling through the first 5 channels of the first subset, essentially performing maximum value processing to output a pixel semantic feature μ. (i，j) Therefore, for a channel set with a channel scale of 4, we can ultimately obtain 4 pixel semantic features.
[0091] In addition, this invention also provides a cross-channel pyramid pooling system, including: a channel scale setting unit, a channel set acquisition unit, a feature map acquisition unit, and a cross-channel pooling unit. The channel scale setting unit is used to set multiple different channel scales, each corresponding to a different cross-channel pooling operation; the channel set acquisition unit is used to acquire multiple channels at each channel scale to obtain multiple channel sets; the feature map acquisition unit is used to acquire multi-layer feature maps, which are obtained by processing the input image through convolutional layers; the cross-channel pooling unit is used to perform cross-channel pooling operations on the multi-layer feature maps at each of the multiple channel sets to obtain the pooling result.
[0092] Specifically, in conjunction with reference Figure 2 and Figure 3 It is understood that the cross-channel pyramid pooling system of this application embodiment is used to implement the cross-channel pyramid pooling method. The cross-channel pyramid pooling system of this application embodiment corresponds to the aforementioned cross-channel pyramid pooling method. For the specific processing process, please refer to the aforementioned cross-channel pyramid pooling method, which will not be repeated here.
[0093] In this embodiment, the spatial pyramid pooling method is improved by incorporating the principle of cross-channel pooling, thereby obtaining the cross-channel pyramid pooling system of this invention. This cross-channel pyramid pooling system extracts pixel semantic features at different channel scales for the same pixel location along the channel dimension, considering the distribution differences of pixel semantic features between channels. This helps to highlight important pixel locations and overcomes the shortcomings of existing spatial pyramid pooling methods. Therefore, the cross-channel pyramid pooling system of this invention can be applied to neural network architectures, such as convolutional neural networks, Transformers, and multilayer perceptrons (MLPs), to improve performance. Furthermore, it can be used for two-dimensional and three-dimensional image classification, image segmentation, object detection and localization, and other computer vision tasks. Simultaneously, by embedding it into a convolutional neural network framework as a main module, it can be deployed to servers, workstations, supercomputing centers, cloud computing centers, personal computers, mobile phones, edge devices, and medical devices.
[0094] Additionally, refer to Figure 4 The present invention also provides a convolutional neural network, including an input layer, a convolutional layer, a pooling layer, a cross-channel pyramid pooling layer, a fully connected layer, and an output layer. The cross-channel pyramid pooling layer is used to perform the cross-channel pyramid pooling method as described in the present invention.
[0095] Specifically, refer to Figure 4 ,by Figure 4 For example, it can be understood that the embodiments of the present invention encapsulate the cross-channel pyramid pooling method into a pyramid method module, that is, to obtain a cross-channel pyramid pooling layer, and insert it into some convolutional neural networks, such as the VGGNet11 network, as a basic component of modern convolutional neural networks to construct a cross-channel pyramid convolutional neural network.
[0096] Furthermore, in conjunction with references Figure 4First, VGGNet11 uses conventional convolutional and pooling layers to extract high-level features of the input image. Then, it uses cross-channel pyramid pooling layers to extract pixel semantic features at different scales along the channel dimension to obtain pixel semantic feature data. Finally, it uses GAP to extract global features of the pixel semantic feature data and inputs them into the classifier Softmax to finally output the prediction results of the convolutional neural network.
[0097] In this embodiment, by utilizing the convolutional neural network of this invention, optimized output results can be obtained. The optimization effect is specifically reflected in the use of a cross-channel pyramid pooling layer. By extracting pixel semantic features at different channel scales for the same pixel location along the channel dimension, the distribution differences of pixel semantic features across channels are considered, which helps to highlight important pixel locations and overcomes the shortcomings of existing spatial pyramid pooling methods. Therefore, the convolutional neural network of this invention can be deployed in practical applications such as image classification, object detection, and segmentation, and can achieve better prediction results.
[0098] In addition, refer to Figure 5 This invention also provides a convolutional neural network processing method, applied to the convolutional neural network of this invention, comprising the following steps:
[0099] The input image data is processed using the input layer, convolutional layer, and pooling layer to obtain high-level feature data, which represents the high-level features extracted from the image.
[0100] High-level feature data is processed using a cross-channel pyramid pooling layer to obtain pixel semantic feature data, which represents the cross-channel pyramid pooling result.
[0101] Fully connected layers and output layers are used to process pixel semantic feature data to obtain the prediction results of convolutional neural networks.
[0102] Specifically, in conjunction with reference Figure 4 and Figure 5 It is understood that the convolutional neural network processing method of this application embodiment is used to execute in the convolutional neural network of this application embodiment. The convolutional neural network processing method of this application embodiment corresponds to the aforementioned convolutional neural network. For the specific processing process, please refer to the aforementioned convolutional neural network, which will not be repeated here.
[0103] In this embodiment, by applying the convolutional neural network processing method of this invention to the convolutional neural network of this invention, optimized output results can be obtained. The optimization effect is specifically reflected in the use of a cross-channel pyramid pooling layer. By extracting pixel semantic features at different channel scales for the same pixel location along the channel dimension, the distribution differences of pixel semantic features between channels are considered, which helps to highlight important pixel locations and overcomes the shortcomings of existing spatial pyramid pooling methods. Therefore, by applying the convolutional neural network processing method of this invention to practical applications such as image classification, object detection, and segmentation, better prediction results can be obtained.
[0104] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions. These instructions are executed by one or more control processors, causing the one or more control processors to perform a cross-channel pyramid pooling method or a convolutional neural network processing method as described above, for example, executing the methods described above. Figure 2 or Figure 5 The method in the middle.
[0105] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0106] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A cross-channel pyramid pooling method, characterized in that, The method comprises the following steps: a plurality of different channel scales are set, and the plurality of different channel scales correspond to a plurality of different cross-channel pooling operations respectively; a plurality of channels under each channel scale are obtained to obtain a plurality of channel sets; a plurality of feature maps are obtained, which are obtained by processing an input image through a convolution layer; cross-channel pooling operations are respectively performed on the plurality of feature maps under the plurality of channel sets to obtain pooling results; wherein the first N The channel dimension corresponding to the channel set is , N is a positive integer; and the cross-channel pooling operation is performed on the multi-layer feature map under each of the channel sets, including the following steps: The multi-layer feature map is divided into M pixel regions, M The pixel regions collectively constitute the multi-layer feature map, M is a positive integer; The first pixel region is subjected to a cross-channel pooling operation under a first channel set to obtain a first number of pixel semantic features. N The first pixel region is subjected to a cross-channel pooling operation under a first channel set to obtain a first number of pixel semantic features. The first pixel region is subjected to a cross-channel pooling operation under a first channel set to obtain a Continue with the remaining Each pixel region is in N Perform cross-channel pooling operations on the aforementioned channel set to obtain Number of pixels semantic features; To merge a quantity of pixel semantic features to obtain pixel semantic feature data, the pixel semantic feature data representing a cross-channel pyramid pooling result.
2. The cross-channel pyramid pooling method of claim 1, wherein, the cross-channel pooling operation adopts a mean cross-channel pooling operation.
3. The cross-channel pyramid pooling method of claim 2, wherein, The mean cross-channel pooling operation is constrained by the following mathematical model: ， ， wherein, W is the number of channels for each channel set; represents the first N channel dimension corresponding to the channel set; represents the first k pixel region, k is a positive integer and is not greater than K ; represents the pixel semantic feature.
4. The cross-channel pyramid pooling method of claim 1, wherein, the cross-channel pooling operation adopts a maximum cross-channel pooling operation.
5. The cross-channel pyramid pooling method of claim 4, wherein, The maximum cross-channel pooling operation is constrained by the following mathematical model: ； ； wherein, W is the number of channels for each channel set; represents the first N channel scale corresponding to the channel set; represents the first k pixel region, k is a positive integer and is not greater than K ; represents the pixel semantic feature.
6. A cross-channel pyramid pooling system, comprising: The method is applied to the cross-channel pyramid pooling method according to any one of claims 1 to 5, and comprises: a channel scale setting unit, configured to set a plurality of different channel scales, and the plurality of different channel scales correspond to a plurality of different cross-channel pooling operations respectively; a channel set obtaining unit, configured to obtain a plurality of channels under each channel scale to obtain a plurality of channel sets; a feature map obtaining unit, configured to obtain a plurality of feature maps, which are obtained by processing an input image through a convolution layer; a cross-channel pooling unit, configured to perform cross-channel pooling operations on the plurality of feature maps under the plurality of channel sets respectively to obtain pooling results.
7. A method for processing a convolutional neural network, the method comprising: The method is applied to a convolutional neural network, which comprises an input layer, a convolution layer, a pooling layer, a cross-channel pyramid pooling layer, a full connection layer, and an output layer, the cross-channel pyramid pooling layer is configured to perform the cross-channel pyramid pooling method according to any one of claims 1 to 5, and the convolutional neural network processing method comprises the following steps: the input image data is processed by using the input layer, the convolution layer, and the pooling layer to obtain high-level feature data, and the high-level feature data represents high-level features extracted from the image; the high-level feature data is processed by using the cross-channel pyramid pooling layer to obtain pixel semantic feature data, and the pixel semantic feature data represents a cross-channel pyramid pooling result; the pixel semantic feature data is processed by using the full connection layer and the output layer to obtain a prediction result of the convolutional neural network.
8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and the computer executable instructions are used to make the computer execute the cross-channel pyramid pooling method according to any one of claims 1 to 5 or the convolutional neural network processing method according to claim 7.