Data processing method and device, electronic equipment, storage medium and chip

Abstract

The present disclosure provides a data processing method, device, electronic equipment, storage medium and chip. The method comprises: obtaining first features of to-be-processed data extracted by each convolution layer in a plurality of ResNet convolution layers, to obtain a plurality of first features; for each first convolution layer, fusing the first feature of the first convolution layer and the corresponding sampling feature to obtain the corresponding fused feature; the sampling feature is obtained by downsampling the fused feature of the previous convolution layer; the first convolution layer is the Nth convolution layer in the plurality of convolution layers, and N is an integer greater than 1; and the fused feature of the last convolution layer in the plurality of convolution layers is taken as feature information of the to-be-processed data. The present disclosure provides a scheme, which can improve the feature extraction accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and chip. Background Technology

[0002] Residual Networks (ResNet) are a type of deep convolutional neural network that solves the vanishing gradient problem in deep learning networks by introducing residual connections. These residual connections establish direct connections between the input and output of the ResNet feature extraction module. In ResNet applications, after the convolutional layers extract precise features, these features are input into the next convolutional layer for further feature extraction. The output of the final convolutional layer is the extracted feature information.

[0003] However, directly extracting features based on the output of the previous convolutional layer results in low feature extraction accuracy. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, electronic device, storage medium, and chip that can improve feature extraction accuracy.

[0005] A first aspect of this disclosure provides a data processing method, the method comprising:

[0006] The first feature of the data to be processed is obtained from each of the multiple convolutional layers in ResNet, resulting in multiple first features;

[0007] For each first convolutional layer, the first feature of the first convolutional layer and the corresponding sampled feature are fused to obtain the corresponding fused feature; the sampled feature is obtained by downsampling the fused feature of the previous convolutional layer; the first convolutional layer is the Nth convolutional layer among the plurality of convolutional layers, where N is an integer greater than 1;

[0008] The fusion features of the last convolutional layer among the multiple convolutional layers are used as the feature information of the data to be processed.

[0009] The method in the above scheme further includes:

[0010] For each first convolutional layer, the fused features of the first convolutional layer are downsampled to obtain the sampled features of the next convolutional layer.

[0011] In the above scheme, downsampling the fused features of the first convolutional layer to obtain the sampled features of the next convolutional layer includes:

[0012] The fused features of the first convolutional layer are downsampled using a max pooling layer to obtain the sampled features of the next convolutional layer.

[0013] The method in the above scheme further includes:

[0014] The first feature of the second convolutional layer is downsampled to obtain the sampling result; the second convolutional layer is the first convolutional layer among the plurality of convolutional layers.

[0015] The sampling results are used as the sampling features of the next convolutional layer.

[0016] In the above scheme, downsampling the first feature of the second convolutional layer to obtain the sampling result includes:

[0017] The first feature of the second convolutional layer is downsampled using an average pooling layer (AvgPool) to obtain the sampling result.

[0018] In the above scheme, fusing the first feature of the first convolutional layer and the corresponding sampled feature to obtain the corresponding fused feature includes:

[0019] Based on the multi-path self-attention mechanism, the first feature and the corresponding sampled feature of the first convolutional layer are fused to obtain the corresponding fused feature.

[0020] The method in the above scheme further includes:

[0021] The feature information is classified to obtain classification information.

[0022] In the above scheme, the classifier includes an activation function and a DropOut function; wherein, the activation function is used to perform non-linear processing on the feature information; and the DropOut function is used to perform regularization processing on the feature information.

[0023] A second aspect of this disclosure provides a data processing apparatus, the apparatus comprising:

[0024] The acquisition unit is used to acquire the first feature of the data to be processed extracted by each of the multiple convolutional layers in ResNet, and obtain multiple first features;

[0025] The sampling fusion unit is used to fuse the first feature of the first convolutional layer and the corresponding sampling feature for each first convolutional layer to obtain the corresponding fused feature; the sampling feature is obtained by downsampling the fused feature of the previous convolutional layer; the first convolutional layer is the Nth convolutional layer among the plurality of convolutional layers, where N is an integer greater than 1;

[0026] The processing unit is used to take the fused features of the last convolutional layer among the multiple convolutional layers as the feature information of the data to be processed.

[0027] A third aspect of this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure, or to perform the methods described in the second aspect of this disclosure.

[0028] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause a computer to perform the methods described in the first aspect of this disclosure, or to perform the methods described in the second aspect of this disclosure.

[0029] A fifth aspect of this disclosure provides a chip including one or more interfaces and one or more processors; the interfaces are configured to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processor, cause the electronic device to perform the methods described in the first aspect of this disclosure, or to perform the methods described in the second aspect of this disclosure.

[0030] In summary, the data processing method, apparatus, electronic device, storage medium, and chip proposed in this disclosure obtain the first features of the data to be processed extracted from each of the multiple convolutional layers in a ResNet, resulting in multiple first features; for each first convolutional layer, the first features of the first convolutional layer and the corresponding sampled features are fused to obtain a corresponding fused feature; the sampled feature is obtained by downsampling the fused feature of the previous convolutional layer; the first convolutional layer is the Nth convolutional layer among the multiple convolutional layers, where N is an integer greater than 1; and the fused feature of the last convolutional layer among the multiple convolutional layers is used as the feature information of the data to be processed. The scheme proposed in this disclosure, by fusing the outputs of each convolutional layer, can associate the output of each convolutional layer with the feature information of the final output, thereby strengthening the perception of each convolutional layer on the feature extraction results and the gradient transfer of ResNet, and improving the accuracy of the feature extraction results. Furthermore, by downsampling the fusion result of the previous convolutional layer and fusing the sampled result with the features output by the current convolutional layer, it is possible to achieve pairwise fusion of the features output by the convolutional layers from shallow to deep, reducing the feature loss of downsampling, so that the feature extraction results output by ResNet can reflect multi-scale feature information and improve the accuracy of the feature extraction results.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0033] Figure 1a This is a schematic diagram of the structure of a feature extraction module in ResNet, a related technology.

[0034] Figure 1b This is a schematic diagram of the structure of another feature extraction module in ResNet, which is related to the technology.

[0035] Figure 2 This is a schematic flowchart of a data processing method provided in an embodiment of the present disclosure;

[0036] Figure 3 A schematic flowchart illustrating feature fusion in the data processing method provided in this embodiment of the disclosure;

[0037] Figure 4 A schematic diagram illustrating the process of classifying final feature information using a classifier in the data processing method provided in this embodiment of the disclosure;

[0038] Figure 5 This is a schematic diagram of another data processing method provided in an embodiment of the present disclosure;

[0039] Figure 6 A classification system architecture diagram based on a multi-path self-attention mechanism is provided for an application embodiment of this disclosure;

[0040] Figure 7 A schematic diagram of the feature fusion process in a classification method based on a multi-path self-attention mechanism provided for an application embodiment of this disclosure;

[0041] Figure 8 This is a schematic diagram comparing the test results of ResNet50 in related technologies with those in the present disclosure application embodiments;

[0042] Figure 9 This is a schematic diagram of the data processing apparatus structure provided in an embodiment of the present disclosure;

[0043] Figure 10 This is a schematic diagram of the electronic device structure provided in the embodiments of this disclosure;

[0044] Figure 11 This is a schematic diagram of the chip structure provided in an embodiment of this disclosure. Detailed Implementation

[0045] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0046] With the rapid development of the internet, a massive amount of information has accumulated, leading to the need for information filtering. Artificial intelligence (AI) technology can enable rapid and accurate information filtering. Image recognition is a widely used AI technology, frequently applied in target detection and segmentation scenarios, such as facial recognition, optical character recognition (ORC), remote sensing image recognition, and machine vision.

[0047] The foundation of image recognition networks is Convolutional Neural Networks (CNNs). However, CNNs suffer from network degradation as the number of layers increases; that is, the accuracy on both the training and test sets decreases with increasing depth. Therefore, CNNs tend to have relatively shallow layers. To address this limitation, ResNet was introduced. ResNet reduces the negative impact of network depth by establishing direct connections between the input and output of its feature extraction module, allowing for an effective depth of up to 152 layers. The feature extraction module is the fundamental module of ResNet, primarily consisting of two types: basic blocks that build shallow networks, such as... Figure 1a As shown, another approach is to construct bottleneck blocks for deep networks, such as... Figure 1b As shown.

[0048] The following description uses ResNet50 as an example to illustrate the overall structure of ResNet. Table 1 shows a list of schematic diagrams of the ResNet50 network structure:

[0049] Output feature shape Network layer name ResNet50 112×112×64 Preprocessing conv+bn+ReLU 56×56×256 Layer-1 maxpool, Bottleneck×3 28×28×512 Layer-2 Bottleneck x4 14×14×1024 Layer-3 Bottleneck x 6 7×7×2048 Layer-4 Bottleneck x3 1×1×1000 Classifier avgpool,fc

[0050] Table 1

[0051] As shown in Table 1, the ResNet50 network structure consists of three parts. The first part is a preprocessing layer composed of convolutional kernels, batch normalization (BN), and rectified linear units (ReLU), used to generate channel-level features of the input image. The second part is the core feature extraction layer, comprising four layers from Layer-1 to Layer-4. Each layer consists of several bottlenecks. During feature extraction in each layer, the input features are downsampled and then processed again by each bottleneck to obtain the output features of that layer. These output features are then input into the next layer to repeat the feature extraction process. The third part is a classifier composed of a global average pooling layer and fully connected layers. The output features of Layer-4 are input into the classifier for final classification. It can be seen that in ResNet, the feature data flow between different network layers is a single, top-down flow.

[0052] In related technologies, three main improvement approaches have been proposed to enhance ResNet performance: First, preserving the overall network structure while improving the feature extraction module. For example, ResNext, based on group convolution, divides features into different groups and extracts features within each group, making the extracted features more biased. Another example is Res2Net, based on multiple receptive field features, which generates features from multiple receptive fields by summing the grouped features one by one within the extraction module. Yet another example is SKNet, based on optional multi-scale feature fusion, which generates multi-scale features through convolutional layers of different kernel sizes and then uses a multi-path self-attention mechanism for feature fusion. Additionally, there is ResNetV2. There are three main approaches to ResNet: 1) Improvements to the feature extraction module, such as ResNet and ResNest; 2) Restructuring the network using the ResNet concept; for example, DenseNet significantly reduces gradient propagation by building dense connections between different network layers; 3) Combining ResNet with other network structures; for example, SE-ResNet with its squeeze-and-excitation (SE) attention module, Ghost-ResNet with its lightweight network structure, and Inception-ResNet combined with the Inception series.

[0053] However, as can be seen from the above description, in existing residual networks derived from ResNet, each layer extracts more accurate features and then passes them to the next layer for final classification. In such a network structure, since the features output by shallow layers directly affect the feature extraction of the next layer, once the feature output of shallow layers has an error, it will directly increase the difficulty for deep layers to extract the correct features. As the feature data flows from shallow to deep from top to bottom along the network layers, the deviation from the accurate features will become larger and larger, resulting in lower accuracy of the final output feature extraction results.

[0054] Based on this, in the various embodiments of this disclosure, by fusing the output results of each convolutional layer, the output of each convolutional layer can be associated with the feature information of the final output, thereby strengthening the perception of each convolutional layer on the feature extraction results and the gradient transfer of ResNet, and improving the accuracy of the feature extraction results. Furthermore, by downsampling the fusion result of the previous convolutional layer and fusing the sampled result with the features output by the current convolutional layer, the features output by the convolutional layers can be fused pairwise from shallow to deep, reducing the feature loss of downsampling, so that the feature extraction results output by ResNet can reflect multi-scale feature information and improve the accuracy of the feature extraction results.

[0055] Figure 2 A flowchart illustrating a data processing method is provided, which can be applied to electronic devices, specifically personal computers (PCs), servers, and other electronic devices. Figure 2 As shown, the method may include:

[0056] Step 201: Obtain the first feature of the data to be processed extracted by each convolutional layer in the ResNet, and obtain multiple first features.

[0057] In practical applications, the ResNet feature extraction layer includes multiple convolutional layers. When the preprocessed data is input into the feature extraction module, the feature extraction module will extract features from the preprocessed data. During the feature extraction process, each convolutional layer will extract features from the input features and input the extracted features into the next convolutional layer. The input features of the first convolutional layer are the preprocessed data. The convolutional layer can also be called a Layer, and this embodiment does not limit it, as long as it can achieve its function.

[0058] In practical applications, the first feature can also be referred to as the output feature of the convolutional layer. This embodiment does not limit this, as long as its function can be achieved.

[0059] Step 202: For each first convolutional layer, fuse the first feature of the first convolutional layer and the corresponding sampled feature to obtain the corresponding fused feature; the sampled feature is obtained by downsampling the fused feature of the previous convolutional layer; the first convolutional layer is the Nth convolutional layer among the plurality of convolutional layers, where N is an integer greater than 1.

[0060] In one embodiment, the method may further include:

[0061] For each first convolutional layer, the fused features of the first convolutional layer are downsampled to obtain the sampled features of the next convolutional layer.

[0062] In practical applications, apart from the first convolutional layer, the features output by each convolutional layer can be further fused with the fused features of the previous layer. Specifically, for the first feature of each convolutional layer, the fused features of the previous convolutional layer are downsampled to obtain sampled features. Then, the sampled features are fused with the first feature of the convolutional layer to generate the fused features of the convolutional layer. The above process is used to fuse the first features of each convolutional layer in turn until the fused features of the last convolutional layer are generated. The fused features of the last convolutional layer are the final feature extraction result.

[0063] In practical applications, for the first convolutional layer, the output features of that convolutional layer can be directly downsampled.

[0064] Based on this, in one embodiment, the method may further include:

[0065] The first feature of the second convolutional layer is downsampled to obtain the sampling result; the second convolutional layer is the first convolutional layer among the plurality of convolutional layers.

[0066] The sampling results are used as the sampling features of the next convolutional layer.

[0067] For example, the output features of each convolutional layer can be represented using the following formula:

[0068]

[0069] Among them, X i X is the feature output by the i-th layer. i ' represents the result of fusing the output features of the i-th layer with those of the upper layer, where Fusion is the feature fusion function and Down is the downsampling function.

[0070] Since CNNs allow for a variety of downsampling functions, for example, most networks perform a global average pooling operation after all feature extraction is completed, reducing the feature map size to 1x1, or a downsampling pooling operation is performed after the preprocessing layer to reduce the computational cost of the network; however, the sampling methods in related techniques do not have a large sampling amplitude because large-scale downsampling can blur the category information of image features. This disclosure minimizes the decrease in the output features of each convolutional layer by sampling the fusion result of each convolutional layer and fusing the sampling result with the output features of the next convolutional layer.

[0071] In practical applications, shallow features have the characteristics of a smaller receptive field, being close to the input end, and containing fine-grained information (such as edge information, color information, corner information, etc.), while deep features have the characteristics of a larger receptive field, being close to the output end, and containing more abstract category information. Different downsampling methods also have different scanning methods for features. Therefore, different sampling methods can be configured for different convolutional layers.

[0072] Based on this, in one embodiment, MaxPool can be used to downsample the fused features of the first convolutional layer to obtain the sampled features of the next convolutional layer.

[0073] In one embodiment, AvgPool can also be used to downsample the first feature of the second convolutional layer to obtain the sampling result.

[0074] In practical applications, when using MaxPool to scan feature maps, the maximum feature value within the scanning range is taken, while using AvgPool is taken as the average value of all features within the scanning range. When the information contained in the fused features of different convolutional layers is different, using MaxPool for downsampling in shallow layers and AvgPool for downsampling in deep layers can achieve fewer trainable parameters and simpler calculations, and can avoid putting too much burden on gradient propagation during backpropagation of the network.

[0075] In one embodiment, after sampling the convolutional layer using MaxPool or AvgPool, the sampled features can be processed by point convolution.

[0076] In one embodiment, the first convolutional layer and the second convolutional layer can use the same downsampling method. Specifically, AvgPool can be used to downsample the first convolutional layer and the second convolutional layer.

[0077] In practical applications, the category information of some features obtained after downsampling may be obscured by the model, thus affecting the accuracy of the final feature output. Therefore, attention-based fusion algorithms can be used in conjunction with downsampling to reduce the negative impact caused by the obscuration of some feature category information by assigning low weights.

[0078] Based on this, in one embodiment, fusing the first feature of the first convolutional layer and the corresponding sampled feature to obtain the corresponding fused feature may include:

[0079] Based on the multi-path self-attention mechanism, the first feature and the corresponding sampled feature of the first convolutional layer are fused to obtain the corresponding fused feature.

[0080] Here, by fusing shallow and deep features, the tendency of image features to deviate from accurate features as the network deepens can be reduced. To avoid the problem of some effective information being blurred due to downsampling and negatively impacting deep features, a self-attention mechanism is introduced on the basis of the fusion algorithm. Due to the introduction of the self-attention mechanism, different weights can be assigned to feature maps at the channel level, so that the network can consciously reduce the influence of erroneous feature maps on the fused features and increase the weight of accurate features. In this embodiment, a multi-path self-attention mechanism, such as SKNet, is used. The multi-path self-attention mechanism can also be called a multi-path self-attention feature fusion mechanism, which is not limited in this embodiment.

[0081] For example, Figure 3 This is a flowchart illustrating the feature fusion process, such as... Figure 3 As shown, during the fusion process, features Figure X i With X i ' -1 First, global average pooling is performed separately, and then the results are summed. The sum can be expressed using the following formula:

[0082]

[0083] Then, X g-avg X is obtained after a fully connected operation. fc1 Then X fc1 X is obtained through a fully connected operation. fc2-1 and X fc2-2 ;X fc2-1 and X fc2-2 The weights are normalized using the softmax function to values ​​within (0, 1) and sum to 1; X fc2-1 The softmax function can be expressed using the following formula:

[0084]

[0085] Where, let z i For X fc2-1 The i-th value in the equation, C is X fc2-1 Length;

[0086] Thus, X can be obtained through the above process. fc2-1 and X fc2-2 The corresponding weights are then assigned to X. i With X i ' -1 Then, by adding them together, the feature fusion can be completed.

[0087] Step 203: Use the fusion feature of the last convolutional layer among the multiple convolutional layers as the feature information of the data to be processed.

[0088] In practical applications, after the feature fusion of the last convolutional layer is completed, the fused features of the last convolutional layer are the final extracted feature information.

[0089] In one embodiment, the method may further include:

[0090] The feature information is classified to obtain classification information.

[0091] In practical applications, the classifier is used to perform the final classification of the fused features; the classifier can also be called a classification layer, and this embodiment does not limit it, as long as it can achieve its function.

[0092] In one embodiment, the classifier may include an activation function and a DropOut function; wherein the activation function is used to perform non-linear processing on the feature information; and the DropOut function is used to perform regularization processing on the feature information.

[0093] In practical applications, the activation function can be ReLU, selu, pReLU, elu, etc., and this disclosure does not limit it.

[0094] For example, Figure 4 This is a flowchart illustrating the process of classifying the final feature information using a classifier; for example... Figure 4 As shown, the ReLU activation function is first used to perform non-linear processing on the final feature information, then the DropOut function is used to regularize the data, and finally the data is passed through a fully connected layer (fc) to obtain the final classification result. The activation function's processing of the feature information can be expressed by the following formula:

[0095] f(x) = max(0,x) (4).

[0096] Here, ResNet uses a combination of convolutional layers, ReLU, and BN functions to form the basic network layers, which reduces the vanishing gradient problem and improves the network's generalization ability. This is a result that is compatible with its residual structure. Feature fusion involves a large number of linear fully connected operations, which reduces the nonlinearity of the model to some extent. The embodiments disclosed in this disclosure improve the nonlinearity of the network by adding activation functions, thereby solving the problems of vanishing and exploding gradients and improving the nonlinearity of the model. By adding dropout, the network can randomly ignore some neurons with a certain probability during training, making the network less dependent on certain local features, thereby improving the network's generalization ability and solving the problem of network overfitting.

[0097] It should be noted that the method provided in this disclosure can be used to improve the classifier in ResNet. The method in this disclosure can also be applied to transfer learning scenarios. Specifically, the trained ResNet classifier is replaced with the improved classifier using this disclosure, and then transferred to other tasks. At the same time, the method provided in this disclosure can be applied to ResNet and its similar residual networks, as well as to other CNNs besides ResNet, such as the Inception series, EfficientNet, MobileNet series, etc., as long as the applied CNN can implement the method provided in this disclosure. This disclosure does not impose any specific limitations.

[0098] In summary, the data processing method provided in this disclosure, by fusing the outputs of each convolutional layer, can associate the output of each convolutional layer with the feature information of the final output, thereby strengthening the perception of each convolutional layer on the feature extraction results and the gradient transfer of ResNet, and improving the accuracy of the feature extraction results. Furthermore, by downsampling the fusion result of the previous convolutional layer and fusing the sampled result with the features output by the current convolutional layer, it is possible to achieve pairwise fusion of the features output by the convolutional layers from shallow to deep, reducing the feature loss of downsampling, so that the feature extraction results output by ResNet can reflect multi-scale feature information and improve the accuracy of the feature extraction results.

[0099] based on Figure 2 The data processing method shown in this disclosure also includes a data processing method in this embodiment. For example... Figure 5 As shown, the method includes:

[0100] Step 501: Obtain the first feature of the data to be processed extracted by each convolutional layer in the ResNet, and obtain multiple first features.

[0101] Step 502: For each first convolutional layer, fuse the first feature of the first convolutional layer and the corresponding sampled feature to obtain the corresponding fused feature; the sampled feature is obtained by downsampling the fused feature of the previous convolutional layer; the first convolutional layer is the Nth convolutional layer among the plurality of convolutional layers, where N is an integer greater than 1;

[0102] Step 503: Use the fusion feature of the last convolutional layer among the multiple convolutional layers as the feature information of the data to be processed.

[0103] Step 504: Classify the feature information to obtain classification information.

[0104] In one embodiment, the method may further include:

[0105] For each first convolutional layer, the fused features of the first convolutional layer are downsampled to obtain the sampled features of the next convolutional layer.

[0106] In one embodiment, downsampling the fused features of the first convolutional layer to obtain the sampled features of the next convolutional layer may include:

[0107] MaxPool is used to downsample the fused features of the first convolutional layer to obtain the sampled features of the next convolutional layer.

[0108] In one embodiment, the method may further include:

[0109] The first feature of the second convolutional layer is downsampled to obtain the sampling result; the second convolutional layer is the first convolutional layer among the plurality of convolutional layers.

[0110] The sampling results are used as the sampling features of the next convolutional layer.

[0111] In one embodiment, downsampling the first feature of the second convolutional layer to obtain the sampling result includes:

[0112] The first feature of the second convolutional layer is downsampled using AvgPool to obtain the sampling result.

[0113] In one embodiment, fusing the first feature and the corresponding sampled feature of the first convolutional layer to obtain the corresponding fused feature includes:

[0114] Based on the multi-path self-attention mechanism, the first feature and the corresponding sampled feature of the first convolutional layer are fused to obtain the corresponding fused feature.

[0115] In one embodiment, the classifier includes an activation function and a DropOut function; wherein the activation function is used to perform non-linear processing on the feature information; and the DropOut function is used to perform regularization processing on the feature information.

[0116] The technical solutions of this disclosure will be further described in detail below with reference to specific application embodiments.

[0117] Figure 6 A classification system architecture diagram based on a multi-path self-attention mechanism is provided for an application embodiment of this disclosure; as follows: Figure 6 As shown,

[0118] The ResNet based on the multi-path self-attention mechanism includes a feature extraction module 601 and a classifier 602; wherein, the classifier 602 includes a sampling fusion module 6021 and a classification layer 6022. The feature extraction module 601, the sampling fusion module 6021 and the classification layer 6022 are described in detail below.

[0119] The feature extraction module 601 includes four layers, from Layer-1 to Layer-4, which are the convolutional layers. After the image passes through the preprocessing layer, it undergoes feature extraction through the four layers. The output features of each layer are passed to the next layer and also to the classifier 602.

[0120] The sampling fusion module 6021 is used to downsample the output features of the shallow layer so that their shape and size are consistent with the output features of the lower layer, and to fuse the two sets of features with the same shape.

[0121] The classification layer 6022 consists of the activation function ReLU, global average pooling GlobalAvgPool, DropOut, and a fully connected layer fc, and is used to classify the feature information output by the fusion module.

[0122] Based on the above architecture diagram, this disclosure also provides a classification method based on a multi-path self-attention mechanism, the method comprising:

[0123] Step 1: Downsampling and fusion, then proceed to Step 2.

[0124] Specifically, the output features of Layer-1 are downsampled, and the downsampled results are fused with the output features of Layer-2 to obtain the fused features of Layer-2; for Layer-3 and Layer-4, the output features of each Layer are fused with the fused features of the previous layer; the fusion results can be found in formula (1).

[0125] Here, during the downsampling process, such as Figure 7 As shown, AvgPool and MaxPool are used in conjunction with point convolution for downsampling; specifically, the first fusion result is downsampled using MaxPool, and the other fusion results are downsampled using AvgPool.

[0126] This disclosure employs a multi-path self-attention feature fusion mechanism for feature fusion in its application embodiments; such as... Figure 3 As shown, during the feature fusion process, features Figure X i With X i ' -1 First, Global AvgPool is performed separately, and then the results are summed; the sum can be found in formula (2); then, X g-avg X is obtained after a fully connected operation. fc1 Then X fc1 X is obtained through a fully connected operation. fc2-1 and X fc2-2 ;X fc2-1 and X fc2-2 The weights are normalized using the softmax function to values ​​within (0, 1) and sum to 1; X fc2-1 The softmax function is given by formula (3); thus, X can be obtained through the above process. fc2-1 and X fc2-2 The corresponding weights are then assigned to X. i With X i ' -1 Then, by adding them together, the feature fusion can be completed.

[0127] Step 2: Categorize.

[0128] Specifically, such as Figure 4 As shown, the ReLU activation function is first used to perform nonlinear processing on the final feature information, then the DropOut function is used to perform regularization on the data, and finally the fully connected layer fc is used to obtain the final classification result; the processing of feature information by the activation function can be found in formula (4).

[0129] Here, activation functions are used to improve the nonlinearity of the network and solve problems such as vanishing and exploding gradients. The activation function used in this application embodiment is ReLU; while the dropout function is used to solve the overfitting problem of the network. It allows the network to randomly ignore some neurons with a certain probability during training, so that the network does not depend on certain local features, thereby improving the network's generalization ability.

[0130] It should be noted that, by enabling the output features of each layer to be directly connected to the classifier in addition to being passed to the next layer, the classifier's perception of all layer features is enhanced, and the gradient propagation of the network is improved. Within the network, the features of each layer are first fused by a feature fusion algorithm based on a multi-path self-attention mechanism before classification. Since the self-attention mechanism allows the classifier to absorb the required features from each layer with weights, when the output features of deeper layers have large deviations, they can also be corrected from shallower features, ensuring the accuracy of the output results.

[0131] The application embodiments disclosed herein have the following advantages:

[0132] Compared to the original ResNet and its derivatives, this application embodiment improves the classifier so that the four layers of the network can be associated with the final output through the classifier. The linear structure of the classifier itself enhances the perception of the classification results by each layer of ResNet and strengthens the gradient propagation of the network. At the same time, the classifier in this application embodiment reduces the feature loss of downsampling by fusing the features output by the layers from shallow to deep, allowing the classifier to combine multi-scale feature information. Furthermore, by using a multi-path self-attention mechanism, the information loss problem of downsampling can be further reduced, allowing the network to select beneficial features for fusion. In addition, this application embodiment improves the generalization ability and nonlinearity of the network after introducing a large number of fully connected operations by introducing additional ReLU and DropOut functions into the classifier.

[0133] The effects of the application embodiments of this disclosure will be explained below with reference to specific experimental results.

[0134] Table 2 is a list of experimental equipment for application embodiments of this disclosure:

[0135] Software List Version operating system Ubuntu 18 programming language Python 3.6 frame PyTorch 1.8 graphics card TitanXp

[0136] Table 2

[0137] The experimental dataset used was Cifar10, which included 10 classes of data, with each class consisting of 5000 training images and 1000 test images. The preprocessing scheme for the experimental dataset was uniform random flipping, random cropping, and standardization. The optimizer used was SGD+momentum with an initial learning rate of 0.1, divided by 10 every 30 epochs, for a total of 120 epochs. The dropout function had a probability p of 0.3. The classifiers of different ResNets were replaced using the classifiers from this application embodiment, and the datasets were trained using the ResNet with the replaced classifier and the original ResNet, respectively. Below, the ResNet with the suffix -CLFFCF represents the ResNet with the replaced classifier. The final training results are shown in Table 3.

[0138]

[0139] Table 3

[0140] As can be seen from Table 3, compared with the original ResNet, the ResNet after replacing the classifier, for example, the ResNet50-CLFFCF after replacing the classifier compared with the original ResNet50, the accuracy of ResNet and its derivative networks has been effectively improved after introducing the classifier of the application embodiment of this disclosure. Moreover, since it mainly uses fully connected operations after global pooling, the additional computation is relatively small. Compared with the original ResNet50, the computation is not significantly increased. That is, the accuracy of the output results is improved while maintaining the original computation. Figure 8 The comparison results of test data of ResNet50 before and after adding CLFFC are shown. Figure 8 It can be seen that the accuracy of ResNet50 after introducing CLFFC on the Cifar10 test set has always remained at a higher level.

[0141] To implement the above data processing method, this disclosure also provides a data processing apparatus, which is installed on an electronic device. For example... Figure 9 As shown, the device 900 may include:

[0142] The acquisition unit 901 is used to acquire the first feature of the data to be processed extracted by each of the multiple convolutional layers in ResNet, and obtain multiple first features;

[0143] The sampling fusion unit 902 is used to fuse the first feature of the first convolutional layer and the corresponding sampling feature for each first convolutional layer to obtain the corresponding fused feature; the sampling feature is obtained by downsampling the fused feature of the previous convolutional layer; the first convolutional layer is the Nth convolutional layer among the plurality of convolutional layers, where N is an integer greater than 1;

[0144] The processing unit 903 is used to take the fusion feature of the last convolutional layer among the multiple convolutional layers as the feature information of the data to be processed.

[0145] In one embodiment, the sampling fusion unit 902 can also be used for:

[0146] For each first convolutional layer, the fused features of the first convolutional layer are downsampled to obtain the sampled features of the next convolutional layer.

[0147] In one embodiment, the sampling fusion unit 902 can be specifically used for:

[0148] MaxPool is used to downsample the fused features of the first convolutional layer to obtain the sampled features of the next convolutional layer.

[0149] In one embodiment, the sampling fusion unit 902 can also be used for:

[0150] The first feature of the second convolutional layer is downsampled to obtain the sampling result; the second convolutional layer is the first convolutional layer among the plurality of convolutional layers.

[0151] The sampling results are used as the sampling features of the next convolutional layer.

[0152] In one embodiment, the sampling fusion unit 902 can be specifically used for:

[0153] The first feature of the second convolutional layer is downsampled using AvgPool to obtain the sampling result.

[0154] In one embodiment, the sampling fusion unit 902 can be specifically used for:

[0155] Based on the multi-path self-attention mechanism, the first feature and the corresponding sampled feature of the first convolutional layer are fused to obtain the corresponding fused feature.

[0156] In one embodiment, the device may further include:

[0157] A classification unit is used to classify the feature information to obtain classification information.

[0158] In one embodiment, the classifier includes an activation function and a DropOut function; wherein the activation function is used to perform non-linear processing on the feature information; and the DropOut function is used to perform regularization processing on the feature information.

[0159] It should be noted that the data processing device provided in the above embodiments is only illustrated by the division of the above program modules during detection. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing device and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0160] Based on the hardware implementation of the above program modules, and in order to implement the method of this disclosure embodiment, this disclosure embodiment also provides an electronic device, such as... Figure 10 As shown, the electronic device 1000 includes:

[0161] Communication interface 1001 enables information exchange with other devices;

[0162] The processor 1002 is connected to the communication interface 1001 to enable information interaction with other devices and to execute the methods provided by one or more of the above-mentioned technical solutions when running a computer program;

[0163] The computer program is stored in the memory 1003.

[0164] Specifically, the processor 1002 can be used for:

[0165] The first feature of the data to be processed is obtained from each of the multiple convolutional layers in ResNet, resulting in multiple first features;

[0166] For each first convolutional layer, the first feature of the first convolutional layer and the corresponding sampled feature are fused to obtain the corresponding fused feature; the sampled feature is obtained by downsampling the fused feature of the previous convolutional layer; the first convolutional layer is the Nth convolutional layer among the plurality of convolutional layers, where N is an integer greater than 1;

[0167] The fusion features of the last convolutional layer among the multiple convolutional layers are used as the feature information of the data to be processed.

[0168] In one embodiment, the sampling processor 1002 can also be used for:

[0169] For each first convolutional layer, the fused features of the first convolutional layer are downsampled to obtain the sampled features of the next convolutional layer.

[0170] In one embodiment, the processor 1002 may specifically be used for:

[0171] MaxPool is used to downsample the fused features of the first convolutional layer to obtain the sampled features of the next convolutional layer.

[0172] In one embodiment, the processor 1002 can also be used for:

[0173] The first feature of the second convolutional layer is downsampled to obtain the sampling result; the second convolutional layer is the first convolutional layer among the plurality of convolutional layers.

[0174] The sampling results are used as the sampling features of the next convolutional layer.

[0175] In one embodiment, the processor 1002 may specifically be used for:

[0176] The first feature of the second convolutional layer is downsampled using AvgPool to obtain the sampling result.

[0177] In one embodiment, the processor 1002 may specifically be used for:

[0178] Based on the multi-path self-attention mechanism, the first feature and the corresponding sampled feature of the first convolutional layer are fused to obtain the corresponding fused feature.

[0179] In one embodiment, the processor 1002 can also be used for:

[0180] A classification unit is used to classify the feature information to obtain classification information.

[0181] In one embodiment, the classifier includes an activation function and a DropOut function; wherein the activation function is used to perform non-linear processing on the feature information; and the DropOut function is used to perform regularization processing on the feature information.

[0182] It should be noted that the specific processing procedure of processor 1002 can be understood by referring to the above method.

[0183] Of course, in practical applications, the various components in electronic device 1000 are coupled together through bus system 1004. It can be understood that bus system 1004 is used to realize the connection and communication between these components. In addition to a data bus, bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 10 The general labeled all buses as Bus System 1004.

[0184] The memory 1003 in this embodiment is used to store various types of data to support the operation of the electronic device 1000. Examples of such data include any computer program used to operate on the electronic device 1000.

[0185] The methods disclosed in the embodiments of this application can be applied to the processor 1002, or implemented by the processor 1002. The processor 1002 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 1002 or by instructions in the form of software. The first processor 1002 mentioned above may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1002 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 1003. The processor 1002 reads the information in the memory 1003 and completes the steps of the aforementioned method in combination with its hardware.

[0186] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0187] It is understood that the memory (memory 1003) in the embodiments of this application can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0188] Embodiments of this disclosure also propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the malware detection method described in the above embodiments of this disclosure.

[0189] Embodiments of this disclosure also provide a computer program product, including a computer program that is executed by a processor using the malware detection method described in the above embodiments of this disclosure.

[0190] Embodiments of this disclosure also propose a chip, which can be found in [reference]. Figure 11 The diagram shows the structure of the chip. Figure 11 The chip shown includes a processor 1101 and an interface circuit 1102. The number of processors 1101 and the number of interface circuits 1102 can be one or more.

[0191] Optionally, the chip also includes a memory for storing necessary computer programs and data; the interface circuit 1102 is used to receive signals from the memory and send signals to the processor 1101, the signals including computer instructions stored in the memory, which, when executed by the processor 1101, cause the electronic device to perform the malware detection method described in the above embodiments of this disclosure.

[0192] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0193] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0194] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0195] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections having one or more wires (control methods), portable computer disk drives (magnetic devices), RAM, ROM, EPROM, or flash memory, fiber optic devices, and portable optical disc read-only memories (CDROMs). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0196] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0197] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0198] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0199] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A data processing method, characterized in that, The method includes: The first feature of the data to be processed is extracted from each of the multiple convolutional layers in the ResNet residual network, resulting in multiple first features; For each first convolutional layer, the first feature of the first convolutional layer and the corresponding sampled feature are fused to obtain the corresponding fused feature; the sampled feature is obtained by downsampling the fused feature of the previous convolutional layer; the first convolutional layer is the Nth convolutional layer among the plurality of convolutional layers, where N is an integer greater than 1; The fusion feature of the last convolutional layer among the multiple convolutional layers is used as the feature information of the data to be processed. The method further includes: For each first convolutional layer, the fused features of the first convolutional layer are downsampled using a max pooling layer (MaxPool) to obtain the sampled features of the next convolutional layer; wherein, the fused features of the first convolutional layer include at least the edge information, color information, and corner information of the image; The method further includes: The first feature of the second convolutional layer is downsampled using an average pooling layer (AvgPool) to obtain the sampling result; the second convolutional layer is the first convolutional layer among the plurality of convolutional layers; wherein, the features of the second convolutional layer include the category information of the image; The sampling results are used as the sampling features of the next convolutional layer.

2. The method according to claim 1, characterized in that, The process of fusing the first feature and the corresponding sampled feature of the first convolutional layer to obtain the corresponding fused feature includes: Based on the multi-path self-attention mechanism, the first feature and the corresponding sampled feature of the first convolutional layer are fused to obtain the corresponding fused feature.

3. The method according to any one of claims 1 to 2, characterized in that, The method further includes: The feature information is classified to obtain classification information.

4. The method according to claim 3, characterized in that, The classifier includes an activation function and a DropOut function; wherein the activation function is used to perform non-linear processing on the feature information; and the DropOut function is used to perform regularization processing on the feature information.

5. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the first feature of the data to be processed extracted by each of the multiple convolutional layers in ResNet, and obtain multiple first features; The sampling fusion unit is used to fuse the first feature of the first convolutional layer and the corresponding sampling feature for each first convolutional layer to obtain the corresponding fused feature; the sampling feature is obtained by downsampling the fused feature of the previous convolutional layer; the first convolutional layer is the Nth convolutional layer among the plurality of convolutional layers, where N is an integer greater than 1; The processing unit is used to take the fusion feature of the last convolutional layer among the multiple convolutional layers as the feature information of the data to be processed; The sampling fusion unit is further used for: For each first convolutional layer, the fused features of the first convolutional layer are downsampled using a max pooling layer (MaxPool) to obtain the sampled features of the next convolutional layer; wherein, the fused features of the first convolutional layer include at least the edge information, color information, and corner information of the image; The sampling fusion unit is further used for: The first feature of the second convolutional layer is downsampled using an average pooling layer (AvgPool) to obtain the sampling result; the second convolutional layer is the first convolutional layer among the plurality of convolutional layers; wherein, the features of the second convolutional layer include the category information of the image; The sampling results are used as the sampling features of the next convolutional layer.

6. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 4.

8. A chip, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processors, cause the electronic device to perform the method of any one of claims 1 to 4.

Images