A data processing method and apparatus

Abstract

This invention provides a data processing method and apparatus. The method includes: generating a salient vector for each group of channels in each convolutional layer based on the input tensor of a first convolutional layer; processing each salient vector to form a decision vector, wherein the decision vector indicates that the first channel in the corresponding group is used for data transmission and the second channel is pruned and eliminated; performing channel pruning based on the decision vector; determining sub-filters corresponding to the first channels of each group based on the decision vectors, thereby obtaining sub-input tensors corresponding to the first channels of each group based on the filters, enabling the image detection model to convolve the sub-input tensors to obtain concatenated output tensors of all groups. This data processing method can accelerate the processing efficiency and improve the processing accuracy of image detection models.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology

[0002] DCNNs have revolutionized the computer vision research community in many applications, from low-level tasks such as saliency detection and edge detection to high-level tasks such as image classification and object detection. The ever-increasing prediction accuracy typically comes at the cost of massive computational demands. How to reduce the computational cost of DCNNs without sacrificing accuracy is a key topic in the vision community.

[0003] Generally, a convolutional layer takes an input tensor And generate the output tensor Where {C, H, W} and {C′, H′, W′} represent the {number of channels, height, and width} of the input and output vectors, respectively, and can be expressed by the following formula:

[0004] Y = [f 1 (X 1 ), f 2 (X 2 ), ..., f G (X G )],

[0005] X i =squeeze(X·π) i (·))

[0006] Where [·] represents a join operation, G represents the number of groups, and f i For convolution operations, and It is a sub-input tensor whose number of channels may be less than X. Select module π. i (·) Generates a vector of length C to determine which channels should be used in the current group i, and a squash operation removes all unselected channels to form a compact subtensor. Many classic frameworks are special cases of this, including:

[0007] When π i (·)=1∈{1} (1×C) This means that all input channels remain in any group, regardless of the group number, and the formula simplifies to normal convolution. In this case, because all input channels are retained, feature redundancy within the network is also preserved, resulting in higher computational cost and relatively higher error. Summary of the Invention

[0008] This invention provides a data processing method and an apparatus for accelerating the processing efficiency and accuracy of image detection models.

[0009] To address the aforementioned technical problems, embodiments of the present invention provide a data processing method applied to an image detection model consisting of multiple consecutive convolutional layers, comprising:

[0010] Based on the input tensor of the first convolutional layer, a salient vector is generated for each group of channels in each convolutional layer;

[0011] Each of the salient vectors is processed to form a decision vector, which indicates that the first channel in the corresponding group is used to transmit data and the second channel is pruned and eliminated.

[0012] Channel pruning is performed based on the decision vector;

[0013] Based on the decision vector, sub-filters corresponding to the first channels of each group are determined, and sub-input tensors corresponding to the first channels of each group are obtained based on the filters, so that the image detection model can convolve the sub-input tensors to obtain the output tensors of all groups after concatenation.

[0014] As an optional embodiment, it also includes:

[0015] The input tensor of the first convolutional layer is downsampled using global average pooling.

[0016] As an optional embodiment, the input tensor based on the first convolutional layer generates a salient vector for each group of channels in each convolutional layer, including:

[0017] Based on the created multilayer perceptron, the input tensor of the first convolutional layer after downsampling is processed, and then a significant tensor is generated for each convolutional layer.

[0018] As an optional embodiment, the multilayer perceptron consists of at least two mapping matrices.

[0019] As an optional embodiment, the process of forming decision vectors from each of the salient vectors includes:

[0020] The decision vectors are formed based on the sine function being located at each of the salient vectors.

[0021] As an optional embodiment, the decision vector corresponding to each convolutional layer is different, and the first channel and the second channel in each convolutional layer are not exactly the same.

[0022] As an optional embodiment, the number of the second channels is at least related to the number of channel groups in the corresponding convolutional layer.

[0023] As an optional embodiment, the correspondence between at least the first channel and the output channel of each convolutional layer includes multiple possibilities.

[0024] Another embodiment of the present invention also provides a data processing apparatus, comprising:

[0025] A generation module is configured to generate a salient vector for each group of channels in each convolutional layer based on the input tensor of the first convolutional layer.

[0026] The first processing module is used to process each of the salient vectors to form a decision vector, wherein the decision vector is used to indicate that the first channel in the corresponding group is used to transmit data and the second channel is pruned and eliminated.

[0027] The determination module is used to determine the sub-filters corresponding to each group of first channels based on the decision vector, so as to obtain the sub-input tensors corresponding to each group of first channels based on the filters, so that the image detection model can convolve the sub-input tensors to obtain the output tensors of all groups after concatenation.

[0028] As an optional embodiment, it also includes:

[0029] The second processing module is used to downsample the input tensor of the first convolutional layer according to global average pooling.

[0030] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0031] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0032] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0033] Figure 1 This is a flowchart of the data processing method in an embodiment of the present invention.

[0034] Figure 2 This is an application diagram of the data processing method in an embodiment of the present invention.

[0035] Figure 3 This is another application diagram of the data processing method in this embodiment of the invention.

[0036] Figure 4 This is another application diagram of the data processing method in this embodiment of the invention.

[0037] Figure 5This is another application diagram of the data processing method in this embodiment of the invention.

[0038] Figure 6 This is another application diagram of the data processing method in this embodiment of the invention.

[0039] Figure 7 This is another application diagram of the data processing method in this embodiment of the invention.

[0040] Figure 8 This is a structural block diagram of the data processing method in an embodiment of the present invention. Detailed Implementation

[0041] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but these are not intended to limit the scope of the invention.

[0042] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this disclosure will be apparent to those skilled in the art.

[0043] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

[0044] These and other features of the invention will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0045] It should also be understood that although the invention has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of the invention, which have the features described in the claims and are therefore all within the scope of protection defined herein.

[0046] The above and other aspects, features and advantages of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0047] Specific embodiments of the present disclosure are described thereafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure and can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the present disclosure. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely to serve as the basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of substantially any suitable detailed structures.

[0048] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in still another embodiment,” all of which may refer to one or more of the same or different embodiments according to this disclosure.

[0049] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0050] Currently, pruning and traditional grouped convolution can effectively reduce the processing load of the model. When performing channel pruning, the device needs to attempt to learn the feature maps that contribute most to the final prediction accuracy, that is, to enable channel pruning to identify unimportant channels that can be pruned without degrading network performance. However, there is currently no good method to guarantee the correctness of this identification, and the representational power of existing pruning rates is limited, so pruning errors will always occur, such as pruning channels containing important information. In particular, larger pruning rates usually lead to worse model performance than smaller pruning rates.

[0051] On the other hand, grouped convolution preserves all input channels, which can avoid the aforementioned information loss problem to some extent. In fact, unlike channel pruning, grouped convolution assumes the network can still learn enough information to give reasonable predictions by uniformly reducing connection density while preserving all input and output channels. However, in this approach, learnability, or the ability to find the optimal combination of input channels to best generate each individual output feature map—in other words, performance—is neglected. That is, poor input channel pruning leads to poor overall model processing efficiency.

[0052] To solve the above problems, such as Figure 1 As shown, this embodiment of the invention provides a data processing method applied to an image detection model consisting of multiple consecutive convolutional layers, including:

[0053] Based on the input tensor of the first convolutional layer, a salient vector is generated for each group of channels in each convolutional layer;

[0054] Each salient vector is processed to form a decision vector, which is used to indicate that the first channel in the corresponding group is used to transmit data, and the second channel is pruned and eliminated.

[0055] Channel pruning based on decision vectors;

[0056] Based on the decision vector, the sub-filters corresponding to the first channel of each group are determined, and the sub-input tensors corresponding to the first channel of each group are obtained based on the filters. This enables the image detection model to convolve the sub-input tensors to obtain the output tensors of all groups after concatenation.

[0057] The method in this embodiment can be applied to object detection models, image detection models, etc., to optimize the "intermediate spectrum" region between channel pruning and grouped convolution, thereby improving the overall processing efficiency and accuracy of the model. The method proposed in this embodiment can be integrated into a module and inserted into the backbone network of the model to reduce the computational cost and load, and improve computational accuracy. The method proposed in this embodiment can be considered as a dynamic routing method, a channel-level dynamic pruning method, and a dynamic filter reweighting method that adjusts the connection topology between channels during model runtime. It can also adjust the use of selected channels, such as the first channel, and determine attention dependent on the data.

[0058] Specifically, in this embodiment, the input tensor of the first convolutional layer is determined. Then, this input tensor is used to calculate and determine the salient vector corresponding to each group of channels in each convolutional layer. These salient vectors are then processed to form corresponding decision vectors. This allows the model to filter multiple channels in the corresponding channel group based on these decision vectors, i.e., to perform channel pruning, to determine the first channel used for data transmission and the second channel that needs to be pruned. Afterward, the model determines sub-filters corresponding to each group of first channels based on the decision vectors, and obtains sub-input tensors corresponding to each first channel based on these sub-filters. Once the model obtains these sub-input tensors, it performs convolution on them, ultimately obtaining the output tensors of all concatenated groups.

[0059] By employing the above method, channels are accurately pruned using decision vectors, reducing the number of useless channels and thus decreasing the model's data processing load, while ensuring the retention of necessary, valuable data. Furthermore, using grouped convolutions to process the retained data further improves the model's data processing efficiency.

[0060] Furthermore, the method in this embodiment also includes:

[0061] The input tensor of the first convolutional layer is downsampled using global average pooling.

[0062] This processing method can reduce the image size and expand the receptive field, facilitating subsequent data processing while reducing computational load.

[0063] Specifically, when generating a salient vector for each group of channels in each convolutional layer based on the input tensor of the first convolutional layer, the process includes:

[0064] Based on the created multilayer perceptron, the input tensor of the first convolutional layer after downsampling is processed, and then a salient tensor is generated for each convolutional layer.

[0065] The multilayer perceptron in this embodiment consists of at least two mapping matrices. Of course, other functional layers can be added to enrich the functionality of the perceptron, and the specifics are not fixed.

[0066] Furthermore, in processing each salient vector to form a decision vector, the following steps are included:

[0067] The decision vectors are formed based on the sine function being located at each salient vector.

[0068] To better describe the method of this embodiment, the following description is provided in conjunction with a specific application:

[0069] Suppose there is a convolutional layer {L} consisting of M consecutive convolutional layers. i The basic building blocks are composed of |i=1,2,…,M} (e.g., the Bottleneck module in ResNet). Convolutional layers L i The number of channels for the input and output tensors are C, respectively. i and C (i+1) The detailed process for constructing the module proposed in this embodiment is as follows:

[0070] Step 1. Assign a group number {G} to each layer. i |i = 1, 2, ..., M}

[0071] Step 2. Generate a salient vector for each group in each layer using the L1 input tensor. Specifically, to keep the module lightweight in this embodiment, global average pooling can be used to first downsample the L1 input tensor to... Then a lightweight multilayer perceptron (MLP) is used to target the layer {L}. i Generate a significant tensor for |i=1,2,…,M} Inspired by the SE module, this embodiment uses two mapping matrices. and The MLP consists of two layers, where R represents the compression ratio. The output of the MLP is then transformed to... This represents the salient vector of the layer (each row represents the salient vector of a certain group). In addition, this embodiment also inserts BN (Batch Normalization) and ReLU (nonlinear activation layer) between the two layers of the MLP, making it a nonlinear transformation.

[0072] Step 3. Use the Sign function

[0073]

[0074] The salient vector R (Gi×Ci)This is converted into a decision vector, indicating which input channels to prune or select in the group, that is, determining the first and second channels. Both the first and second channels include multiple channels; the terms "first channel" and "second channel" are generic terms used here. Typically, in application, 1 can be used to represent "selected" and 0 to represent "pruned".

[0075] Step 4. Select the corresponding filter and input channel, perform convolution on each group, and finally concatenate the output tensors of all groups. Specifically, each convolutional layer L... i The complete filter corresponding to all groups in the array is W∈R (k×k×Ci×(Ci+1) / Gi) , where k represents the size of the convolution kernel. Based on the decision vector of this group, we can proceed along C i The dimensions determine the selected sub-filters, and then the sub-input tensors output by the corresponding selected sub-filters are convolved to form the output tensor of the corresponding group. This is equivalent to grouped convolution. After obtaining the convolution result of each group, i.e., the output tensor of each group, they are concatenated to obtain the final output image. In this embodiment, all filters are stored in the model's memory for selection and learning via gradient descent, which is why the full capabilities of the original network can be preserved.

[0076] Optionally, in step 5, in this embodiment, a parallel MLP can also be added to generate another layer with the target layer L. i The input feature maps in the selected channels are rescaled using the corresponding attention values, with the same salient tensor as the attention vector.

[0077] Combination Figure 2 As shown, the first and second channels are simply described as "trimmed" and "untrimmed". The method of this embodiment can create a richer hierarchical pyramid structure: at the bottom are the canonical channels, which are the basic components of all groups, then upwards are the first channels that are filtered out, which are not unique, and finally at the top are the channels that are completely trimmed and discarded in the current inference.

[0078] Furthermore, in this embodiment, each convolutional layer is grouped, but the grouping strategy is different, for example:

[0079] Grouping granularity: Soft assignment of input channels is used for each group, meaning each input channel has the opportunity to connect to any output group. That is, the correspondence between at least the first channel in each convolutional layer and the output channel of that layer can be multiple. This has at least two advantages: first, no inter-group interaction modules need to be set up; second, an implicit hierarchical pyramid structure of information between input feature maps is built, which can better optimize the connection topology within groups.

[0080] Layer-by-layer granularity: Different layers can have different pruning rates. Unlike the previously used grouped convolution methods, the pruning rate in this embodiment is predefined by the number of groups. That is, each convolutional layer corresponds to a different decision vector, and the first and second channels in each convolutional layer are not completely identical, with the number of second channels being at least related to the number of channel groups in the corresponding convolutional layer. This design can simultaneously possess the functions of channel pruning or network structure search techniques. In application, it can be recognized that each convolutional layer has its own attributes (e.g., shallower and deeper layers can extract different levels of image abstraction), thus requiring different proportions of input channels.

[0081] Per-sample granularity: The connection topology within a group is dynamically generated based on each input sample. By respecting the distinct characteristics of individual samples, this helps improve the adaptability of the model network. Furthermore, it provides better interpretability of the model, as it allows identification of which input channel or group exhibits a specific response to a given network unit for a given sample. Finally, dynamic execution optimizes resource allocation, allowing for more computational resources to be allocated to more difficult samples and vice versa, keeping the overall average computational load low.

[0082] Per-prediction granularity: To prune or select specific channels (gating), a saliency vector / matrix can first be generated, with each element corresponding to a channel. Then, a Top-K function or threshold is used to select the top-ranked channels. Afterward, the saliency values ​​can be reused for attention of the relevant channels. This approach essentially combines gating and attention functions through a single saliency generation to achieve pruning. In this embodiment, gating and attention functions can also be decoupled into two independent dimensions to diversify the decision-making process. As mentioned above, another MLP can be generated by attaching a parallel MLP to the target layer L. i The input feature maps in the selected channels are rescaled using a salient tensor of the same size as the attention vector, and the input feature maps in the selected channels are then rescaled with the relevant attention values. This embodiment can perform this step in parallel with the previous steps, thereby increasing the diversity of the decision-making process.

[0083] Since the only non-differentiable part is the Sign function, which converts continuous salient vectors into binary vectors, this embodiment can use other gradient approximation techniques, such as the pass-through estimator (STE) or Gumbel-Softmax reparameterization, to optimize the entire process end-to-end based on gradient descent. Once the model determines that the above optimization process is learnable, there are no restrictions on groups, layers, and samples, and optional step 5 can be incorporated to further optimize the computational flow.

[0084] Furthermore, the method in this embodiment can be integrated into a module, which can be specifically materialized as a plug-and-play booster, aiming to reduce computational costs while enhancing the original prediction performance of the target backbone network. In implementation, a filter from a regular convolutional layer can also be used. Divided into This serves as the initial state for each filter group when the module is inserted.

[0085] The reduction in computational cost in this embodiment primarily stems from the sparse connections within the groups, and is also attributed to the precedent pruning decisions, i.e., decision vectors, applied to subsequent layers using all input feature maps from the first layer. This means that this embodiment can directly skip the computation of output channels from layers discarded by all groups in the next convolutional layer. Furthermore, the number of MACs in the lightweight MLP is negligible compared to the original model. For example, in ResNet-18, this embodiment uses 1.8 million MACs, which is less than 0.1% of the 1.8 billion MACs in the original model.

[0086] MACs in the backbone network can be calculated using binary decision vectors, and then, budget loss... The network can be regularized under the target computational budget:

[0087]

[0088] in This is the average value of MACs from the current small batch of runs. These are the MACs of the original model, λ is a control variable, and τ is the target residual rate. They can be added to the standard classification loss during model training.

[0089] For other grouping mechanisms, methods such as LGC, FLGC, and DGC can also preserve most or all of the input channels in convolutional layers while benefiting from the learnability of channel partitioning. However, the explorable spectrum of these methods is relatively narrow. For example, although DGC and CGNet are dynamic, they are limited to some extent by the predefined pruning rate of each convolutional layer. FLGC and DGConv actually shuffle the input channels (i.e., restricting each input channel to belong to one and only one group), thus limiting the pruning rate. LGC and SGCNN can only be applied to fixed group topologies. Therefore, in contrast, the method in this embodiment can search for highly optimized structures from the introduced four granularities.

[0090] Furthermore, in practical applications, such as training an image detection model for image classification, a large dataset can be prepared, such as 1.2 million images, with 50,000 images used for validation and 1,000 classes. During training, the data is augmented through random cropping (to a size of 224×224) and random horizontal flipping.

[0091] For the backbone network of the model: the modules used to implement the methods in this embodiment can be inserted into the ResNet-18 / 50, DenseNet, and MobileNetV2 models, as summarized below:

[0092] ResNet-18: The modules in this embodiment are named {G} i The parameters of} = {1, 4} are configured on each basic module, and attention mechanisms can be optionally added to both convolutional layers, as in step 5 of this embodiment.

[0093] ResNet-50: The modules in this embodiment can be in {G i The parameters of} = {1, 4, 1} are configured on each bottleneck module, and an attention mechanism can be optionally added to the second layer.

[0094] DenseNet: You can choose the CondenseNet architecture for the ImageNet dataset, using {G i The module in this embodiment with the parameter configuration of {1,4} replaces the LGC, and an attention mechanism can be optionally added to the second layer.

[0095] MobileNetV2: In the Reverted Bottleneck module, the first expansion layer can remain unchanged. Since the middle 3×3 layers are depth-wise separable, only the last 1×1 layer is configured with the module of this embodiment with G=4, while the input of the lightweight MLP still comes from the input tensor of the first layer. Of course, the attention mechanism can also be selectively attached to the last layer.

[0096] Training Setup: As a plug-in module, this module in this embodiment has the advantage of directly pruning and improving the pre-trained model. For example, the residual rate τ in the loss function is initially set to 1.0, and then gradually reduced to the target residual rate during training. By using a large λ (i.e., 30 in the experiment), the computational cost can be gradually reduced to the budget. The latter part of training can be seen as a fine-tuning process of the model. Specifically, ResNet-18 / 50 and CondenseNet can be trained for 120 epochs each, and MobileNetV2 for 150 epochs, with a batch size of 256. The initial learning rate of the MLP is set to 0.075, and the learning rate of the pre-trained weights in the backbone network is set to 0.015, both decaying to 0 in a cosine shape. Stochastic gradient descent (SGD) with a momentum of 0.9 is used as the optimizer, and the weight decay is set to 10. (-4) This is applied only to the backbone weights. Then, Gumbel-Softmax is used to binarize the saliency tensor. The entire process is implemented using the PyTorch library on two GPUs.

[0097] The method in this embodiment is compared with conventionally used methods, including channel pruning, grouped convolution-based approaches, and the "intermediate spectrum" method. Under the same testing conditions, the test results show that, unlike previous methods, the method in this embodiment can even achieve the desired performance improvement when MACs are reduced. This phenomenon is mainly achieved by the channel reorganization used in this embodiment, such as effectively removing redundant parameters and standardizing the hierarchical reuse of channels.

[0098] Additionally, it should be noted that while the attention mechanism works well on relatively lightweight backbones such as ResNet-18 and MobileNetV2, it may lead to performance degradation or negative effects on large backbones such as DenseNet and ResNet-50, meaning that it may reach saturation under current configurations. Training logs are as follows: Figure 3 As shown in the figure, in the first half of the training, the computational cost steadily decreases to the budget, accuracy increases slowly, and the budget loss remains around 0. In the second half of the training, accuracy is significantly compensated for.

[0099] Optionally, the method in this embodiment can also be applied to object detection models. For example, a large dataset can be prepared, specifically images. During training, the long side of the images is adjusted to 1333, combined with random horizontal flipping. In the validation step, the images are adjusted so that the short side is ≤800 and the long side is ≤1333. In this embodiment, the effect of the module deployed on the ResNet-50 and MobileNetV2 backbones can be evaluated under the Faster R-CNN and RetinaNet frameworks. The backbone network configuration is the same as the experiment on ImageNet. Based on the experiments in the previous section, the module (method) in this embodiment focuses on MobileNetV2, rather than ResNet-50. During training, this embodiment mainly follows the settings on ImageNet and uses the Detectron2 library. After testing the trained model with the same methods as previous models, it can be seen that for ResNet-50, the module in this embodiment can effectively reduce the MACs of the backbone network by 54%, while the accuracy decrease is very small and almost negligible. For MobileNetV2, MSGC achieves significantly better performance than MobileNetV2-0.75 on both detection frameworks with similar computational overhead. Notably, on the RetinaNet framework, the performance of MobileNetV2 can also be improved by inserting MSGC, reducing MACs by 28.5%. This comparison demonstrates that the method in this embodiment also exhibits excellent generalization ability for object detection tasks.

[0100] Furthermore, combined Figure 4 As shown in the figure, this embodiment calculates the probability that each input channel is selected by at least g groups on the ImageNet validation set based on ResNet-18. Each curve is plotted by arranging the probabilities in descending order. The curves from left to right represent g = {4, 3, 2, 1}. Region "A" represents the canonical channels that are continuously used by all groups, and region "B" represents potentially meaningless or redundant discarded channels. Figure 4 As can be seen, Group=1 simplifies the module (method) in this embodiment to a dynamic channel pruning module. Dynamic execution helps to prune network redundancy information of input samples, resulting in better channel discrimination. Meanwhile, setting more groups leads to greater performance improvements for the model, with performance metrics peaking at group=4. This is because a larger number of groups may introduce additional learning complexity, thereby improving model performance.

[0101] Moreover, by Figure 5As can be seen, the method in this embodiment can automatically detect those canonical, generic, and meaningless channels in a hierarchical manner. It is worth noting that the curves in shallower layers appear steeper (meaning that most samples select the same channel), which is consistent with the fact that feature maps in shallow layers typically contain most of the shared low-level or basic image information. In deeper layers, however, the samples may become more dispersed due to differences in semantic features.

[0102] Continue to combine Figure 6 As shown, this figure illustrates the network structure after pruning. The bars represent the number of input channels in each layer of the original model. The middle curve represents the average number of channels used in each group. The top curve represents the residual rate (the quotient of the middle line divided by the bars; a larger value indicates more channels remaining). Based on Figure 6 The dynamic pruning structures of ResNet-50 and MobileNetV2 are revealed, and based on these structures, significant peaks in the residual rate curves can be identified at layers where the image undergoes downsampling. Since reduced resolution shrinks the information in the feature maps, more channels are needed to ensure the representational power of the next layer. In ResNet-50, the last layer of each block also has some minor peaks, expanding the channels for the next Bottleneck module. Therefore, retaining more input channels is expected to preserve more data information.

[0103] At the sample-by-sample granularity, the method in this embodiment, due to its dynamic characteristics, can provide adaptive resource allocation functionality. For example... Figure 7 As shown, the original model and the model equipped with MSGC only achieve good prediction accuracy on "easy" samples. However, the method in this embodiment achieves better predictions because it automatically spends more computation on "difficult" samples (the original model uses more MACs on every sample, while this embodiment selectively uses more MACs on difficult samples). The "easy" and "difficult" samples identified by the method in this embodiment are very intuitive for the user, such as... Figure 7 As can be seen from the explanation at the bottom. For example, the background of the "simple" sample is less cluttered and easier for the human eye to recognize, while the "complex" sample is the opposite (in fact, the sample on the far right is often in black and white style).

[0104] For processing at the decision-level granularity, such as Figure 7 As shown, in Figure 7 In the diagram, shallower layers exhibit relatively fixed gating patterns, meaning that the channels highlighted in the visualized gating matrix are selected by the majority of samples, while deeper layers show greater variation. Channels with gating probabilities close to zero also have attention close to zero. However, despite the fixed gating patterns in shallow layers, additional attention values ​​inject extra variation, enriching the representation of these layers.

[0105] In summary, the method proposed in this embodiment explores a broad "mid-spectral" region between channel pruning and grouped convolution. This method benefits from both aspects and serves as a plug-in module for various backbone networks such as ResNet, DenseNet, and MobileNetV2. Furthermore, the method in this embodiment can reduce the computational cost of these backbone networks by 35% to 50%, while still showing improved accuracy on large-scale ImageNet datasets. Moreover, the method in this embodiment can also be used as a dynamic module, fixed as a plug-in in the model, and it also demonstrates great interpretability for image representations, making it widely applicable.

[0106] like Figure 8 As shown, a data processing apparatus is characterized by comprising:

[0107] A generation module is configured to generate a salient vector for each group of channels in each convolutional layer based on the input tensor of the first convolutional layer.

[0108] The first processing module is used to process each of the salient vectors to form a decision vector, wherein the decision vector is used to indicate that the first channel in the corresponding group is used to transmit data and the second channel is pruned and eliminated.

[0109] The determination module is used to determine the sub-filters corresponding to each group of first channels based on the decision vector, so as to obtain the sub-input tensors corresponding to each group of first channels based on the filters, so that the image detection model can convolve the sub-input tensors to obtain the output tensors of all groups after concatenation.

[0110] As an optional embodiment, the apparatus in this embodiment further includes:

[0111] The second processing module is used to downsample the input tensor of the first convolutional layer according to global average pooling.

[0112] As an optional embodiment, the input tensor based on the first convolutional layer generates a salient vector for each group of channels in each convolutional layer, including:

[0113] Based on the created multilayer perceptron, the input tensor of the first convolutional layer after downsampling is processed, and then a significant tensor is generated for each convolutional layer.

[0114] As an optional embodiment, the multilayer perceptron consists of at least two mapping matrices.

[0115] As an optional embodiment, the process of forming decision vectors from each of the salient vectors includes:

[0116] The decision vectors are formed based on the sine function being located at each of the salient vectors.

[0117] As an optional embodiment, the decision vector corresponding to each convolutional layer is different, and the first channel and the second channel in each convolutional layer are not exactly the same.

[0118] As an optional embodiment, the number of the second channels is at least related to the number of channel groups in the corresponding convolutional layer.

[0119] As an optional embodiment, the correspondence between at least the first channel and the output channel of each convolutional layer includes multiple possibilities.

[0120] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A data processing method applied to an image detection model consisting of multiple consecutive convolutional layers, the image detection model being used for image classification, characterized in that, include: Based on the input tensor of the first convolutional layer, a salient vector is generated for each group of channels in each convolutional layer; Each of the salient vectors is processed to form a decision vector, which indicates that the first channel in the corresponding group is used to transmit data and the second channel is pruned and eliminated. Channel pruning is performed based on the decision vector; Based on the decision vector, sub-filters corresponding to the first channels of each group are determined, and sub-input tensors corresponding to the first channels of each group are obtained based on the filters, so that the image detection model can convolve the sub-input tensors to obtain the output tensors of all groups after concatenation. The input tensor based on the first convolutional layer generates a salient vector for each group of channels in each convolutional layer, including: Based on the created multilayer perceptron, the input tensor of the first convolutional layer after downsampling is processed, and then a significant tensor is generated for each convolutional layer. The connection topology in each channel is dynamically generated based on each input sample.

2. The method according to claim 1, characterized in that, Also includes: The input tensor of the first convolutional layer is downsampled using global average pooling.

3. The method according to claim 1, characterized in that, The multilayer perceptron consists of at least two mapping matrices.

4. The method according to claim 1, characterized in that, The process of forming decision vectors from each of the salient vectors includes: Each of the salient vectors is processed based on a sine function to form the decision vectors respectively.

5. The method according to claim 1, characterized in that, Each convolutional layer has a different decision vector, and the first and second channels in each convolutional layer are not exactly the same.

6. The method according to claim 5, characterized in that, The number of the second channel is at least related to the number of channel groups in the corresponding convolutional layer.

7. The method according to claim 1, characterized in that, The correspondence between at least the first channel and the output channel of each convolutional layer includes multiple possibilities.

8. A data processing apparatus, applied to an image detection model consisting of multiple consecutive convolutional layers, the image detection model being used for image classification, characterized in that, include: A generation module is configured to generate a salient vector for each group of channels in each convolutional layer based on the input tensor of the first convolutional layer. The first processing module is used to process each of the salient vectors to form a decision vector, wherein the decision vector is used to indicate that the first channel in the corresponding group is used to transmit data and the second channel is pruned and eliminated. The determination module is used to determine the sub-filters corresponding to each group of first channels based on the decision vector, so as to obtain the sub-input tensors corresponding to each group of first channels based on the filters, so that the image detection model can convolve the sub-input tensors to obtain the output tensors of all groups after concatenation. The input tensor based on the first convolutional layer generates a salient vector for each group of channels in each convolutional layer, including: Based on the created multilayer perceptron, the input tensor of the first convolutional layer after downsampling is processed, and then a significant tensor is generated for each convolutional layer. The connection topology in each channel is dynamically generated based on each input sample.

9. The data processing apparatus according to claim 8, characterized in that, Also includes: The second processing module is used to downsample the input tensor of the first convolutional layer according to global average pooling.

Images