Deep learning network model, feature extraction method, device and storage medium

Abstract

The present invention relates to the field of deep learning technology, and specifically to a lightweight deep learning network model with an attention mechanism, a feature extraction method, a device and a storage medium. It includes: a basic convolution module, a feature extraction module, a dimension transformation module, and a classification probability output module. Among them, the feature extraction module includes a plurality of feature extraction submodules connected in series. When the network model of this embodiment is used to extract image feature information, a plurality of feature submodules can perform multi-scale spatial feature extraction on the image, so that multiple state information can be extracted at the same time, and the accuracy of feature extraction is improved under the premise of lightweight model.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a lightweight deep learning network model with an attention mechanism, a feature extraction method, a device, and a storage medium. Background Art

[0002] Image attribute recognition is to use machine vision to identify whether the image content has the characteristics and various attributes of interest in a large number of given images. For example, in the field of safe driving, machine vision can be used to learn the content of driver monitoring images, identify the driver's status, and achieve real-time monitoring of safe driving. However, the traditional ResNet (residual network) and GoogLeNet (deep learning structure) classical network models used in the existing technology have large number of calculation parameters and slow speed. Therefore, the speed of extracting image feature information is slow and the accuracy is low, which cannot meet the application requirements of actual scenarios. Summary of the invention

[0003] The main technical problem solved by the present invention is that the traditional network model is slow and has low accuracy when extracting image feature information. Based on this technical problem, the present application proposes a lightweight deep learning network model with an attention mechanism, which aims to improve the speed and accuracy of feature information extraction.

[0004] A lightweight deep learning network model with attention mechanism, which includes: basic convolution module, feature extraction module, dimension transformation module, classification probability output module;

[0005] The input end of the basic convolution module is used to input the image to be processed, the output end of the basic convolution module is connected to the input end of the feature extraction module, the output end of the feature extraction module is connected to the input end of the dimension transformation module, and the output end of the dimension transformation module is connected to the input end of the classification probability output module;

[0006] The basic convolution module is used to perform preliminary feature extraction processing on the processed image to obtain preliminary feature information, and the feature extraction module is used to perform deep feature extraction and fusion on the preliminary feature information to obtain a feature matrix; the dimension transformation module is used to perform dimensionality reduction processing on the feature matrix and transform it into a feature vector that matches the output dimension; the classification probability output module is used to perform probability calculation on the feature vector to obtain the probability corresponding to each state classification;

[0007] Wherein, the feature extraction module includes a plurality of feature extraction submodules connected in series.

[0008] In one embodiment, the feature extraction submodule includes a first convolution module, a second convolution module, a first channel, a second channel, a third channel, a fourth channel, and a first adder;

[0009] The output end of the first convolution module is connected to the input end of the first channel, the second channel, the third channel, and the fourth channel respectively, and the output end of the first channel, the second channel, the third channel, and the fourth channel is connected to the input end of the second convolution module after passing through the first adder;

[0010] Among them, the first channel includes a first attention mechanism module, a first AG module and a first multiplier, the output end of the first attention mechanism module is also connected to the input end of the first AG module, and the output end of the first AG module and the output end of the first attention mechanism module are connected to the first adder after passing through the first multiplier; wherein, the AG module of this embodiment can be understood as an aggregation gate, and the implementation of the AG module is to use global average pooling plus full connection. The AG module is similar to the SE module, but not as complex as the SE module. The final output result of the AG module is the weight value of each information flow.

[0011] The second channel includes a third convolution module, a second attention mechanism module, a second AG module and a second multiplier; the input end of the third convolution module is connected to the output end of the first convolution module, and the output end of the third convolution module is connected to the input end of the second attention mechanism module; the output end of the second attention mechanism module is connected to the input end of the second AG module, and the output end of the second attention mechanism module and the output end of the second AG module are also connected to the first adder after passing through the second multiplier;

[0012] The third channel includes a fourth convolution module, a fifth convolution module, a third attention mechanism module, a third AG module and a third multiplier; the input end of the fourth convolution module is connected to the output end of the first convolution module, and the output end of the fourth convolution module is connected to the input end of the fifth convolution module; the output end of the fifth convolution module is connected to the input end of the third attention mechanism module, the output end of the third attention mechanism module is connected to the input end of the third AG module, and the output end of the third AG module and the output end of the third attention mechanism module are also connected to the first adder after passing through the third multiplier;

[0013] The fourth channel includes a sixth convolution module, a seventh convolution module, an eighth convolution module, a fourth attention mechanism module, a fourth AG module and a fourth multiplier; the input end of the sixth convolution module is connected to the output end of the first convolution module, the output end of the sixth convolution module is connected to the input end of the seventh convolution module, and the output end of the seventh convolution module is connected to the input end of the eighth convolution module; the output end of the eighth convolution module is connected to the input end of the fourth attention mechanism module, the output end of the fourth attention mechanism module is connected to the input end of the fourth AG module, and the output end of the fourth AG module and the output end of the fourth attention mechanism module are connected to the first adder after passing through the fourth multiplier.

[0014] In one embodiment, the first convolution module is used to perform dimension processing on the input feature information to obtain feature information of a target dimension;

[0015] The first channel, the second channel, the third channel, and the fourth channel are respectively used to extract feature information of the target dimension to obtain feature flows of different scales;

[0016] Among them, in the first channel, the first attention mechanism module is used to extract features from the feature information of the target dimension and add the attention mechanism to obtain a first feature stream; the first AG module is used to assign corresponding weights to the first feature stream output by the first attention mechanism module to obtain a first weight feature stream; finally, the first feature stream and the first weight feature stream are multiplied to obtain a first scale feature stream;

[0017] In the second channel, the third convolution module is used to extract features from the feature information of the target dimension to obtain coarse-grained feature information; the second attention mechanism module is used to further extract features from the coarse-grained feature information to obtain a fine-grained second feature stream; the second AG module is used to assign corresponding weights to the second feature stream to obtain a second weighted feature stream; finally, the second feature stream and the second weighted feature stream are multiplied to obtain a second scale feature stream;

[0018] In the third channel, the fourth convolution module and the fifth convolution module are respectively used to extract features of the feature information of the target dimension in sequence, and then the extracted features are further extracted through the third attention mechanism module to obtain a third feature stream; the third AG module is used to assign corresponding weights to the third features to obtain a third weight feature stream; finally, the third feature stream and the third weight feature stream are multiplied to obtain a third scale feature stream;

[0019] In the fourth channel, the sixth convolution module, the seventh convolution module and the eighth convolution module are respectively used to extract features from the feature information of the target dimension in sequence, and then the extracted features are further extracted through the fourth attention mechanism module to obtain a fourth feature stream; the fourth AG module is used to assign corresponding weights to the fourth features to obtain a fourth weight feature stream; finally, the fourth feature stream and the fourth weight feature stream are multiplied to obtain a fourth scale feature stream;

[0020] Adding the first scale feature stream, the second scale feature stream, the third scale feature stream and the fourth scale feature stream through the first adder to obtain the feature streams of different scales;

[0021] The second convolution module is used to adjust the dimensions of the output feature streams of different scales so that their dimensions are the same.

[0022] In one embodiment, it also includes a residual module and a second adder;

[0023] The output end of the second convolution module is connected to the second adder, and the first convolution module is further connected to the second adder after passing through the residual module;

[0024] The input feature information is subjected to residual processing by the residual module and then added to the feature stream of the same dimension output by the second convolution module and then output.

[0025] In one embodiment, the first convolution module and the second convolution module are both 1×1 convolution modules;

[0026] The third convolution module, the fourth convolution module, the fifth convolution module, the sixth convolution module, the seventh convolution module, and the eighth convolution module are all Lite3×3 convolution modules.

[0027] In one embodiment, the first attention mechanism module, the second attention mechanism module, the third attention mechanism module, and the fourth attention mechanism module all include a PSA module (Pyramid Split Attention); the PSA module divides the input tensor (input data) into S groups from the channel. Each group is convolved with different convolution kernel sizes to obtain receptive fields of different scales and extract information of different scales. Then, the SE module is used to extract the weighted values ​​of the channels of each group, and finally the weighted values ​​of the S groups are normalized and weighted by softmax (normalized exponential function).

[0028] The PSA module includes a fifth channel, a sixth channel, a seventh channel, an eighth channel, a first SE module, a second SE module, a third SE module, and a fourth SE module;

[0029] The fifth channel includes a ninth convolution module, the input end of the ninth convolution module is used to input the split feature information, and the output end is connected to the input end of the first SE module;

[0030] The sixth channel includes a tenth convolution module and an eleventh convolution module, the input end of the tenth convolution module is used to input the split feature information, and the output end is connected to the input end of the eleventh convolution module, and the output end of the eleventh convolution module is connected to the input end of the second SE module;

[0031] The seventh channel includes a twelfth convolution module, a thirteenth convolution module, and a fourteenth convolution module; the input end of the twelfth convolution module is used to input the split feature information, the output end of the twelfth convolution module is connected to the input end of the thirteenth convolution module, the output end of the thirteenth convolution module is connected to the input end of the fourteenth convolution module, and the output end of the fourteenth convolution module is connected to the input end of the third SE module;

[0032] The eighth channel includes a fifteenth convolution module, a sixteenth convolution module, a seventeenth convolution module, and an eighteenth convolution module; the input end of the fifteenth convolution module is used to input the split feature information, the output end of the fifteenth convolution module is connected to the input end of the sixteenth convolution module, the output end of the sixteenth convolution module is connected to the input end of the seventeenth convolution module, the output end of the seventeenth convolution module is connected to the input end of the eighteenth convolution module, and the output end of the eighteenth convolution module is connected to the input end of the fourth SE module;

[0033] The output end of the ninth convolution module, the output end of the eleventh convolution module, the output end of the fourteenth convolution module, and the output end of the eighteenth convolution module are connected in sequence;

[0034] The output end of the first SE module, the output end of the second SE module, the output end of the third SE module and the output end of the fourth SE module are connected in sequence.

[0035] In one embodiment, further comprising a fifth multiplier;

[0036] The fifth channel, the sixth channel, the seventh channel, and the eighth channel are respectively used to extract features from the input split feature information to obtain feature streams of different dimensions;

[0037] The first SE module, the second SE module, the third SE module, and the fourth SE module are used to assign weights to the feature streams output by the fifth channel, the sixth channel, the seventh channel, and the eighth channel, respectively;

[0038] The feature streams of different dimensions output by the fifth channel, the sixth channel, the seventh channel, and the eighth channel are spliced, and the spliced ​​feature stream and the spliced ​​weight vector are multiplied by the fifth multiplier to complete the recalibration of the feature stream and obtain multi-scale feature information.

[0039] A method for extracting features from image information, comprising:

[0040] The image information is input into the deep learning network model as described above to obtain multiple feature vectors and the probability corresponding to each feature vector.

[0041] A feature extraction device, comprising:

[0042] A memory for storing a program for constructing a deep learning network model as described above;

[0043] The processor is used to extract features of the image information by executing the program stored in the memory.

[0044] A computer-readable storage medium stores a program, which can be executed by a processor to implement the feature extraction method of image information as described above.

[0045] According to the deep learning network model with attention mechanism of the above embodiment, it includes: basic convolution module, feature extraction module, dimension transformation module, classification probability output module. The input end of the basic convolution module is used to input the image to be processed, the output end of the basic convolution module is connected to the input and output end of the feature extraction module, the output end of the feature extraction module is connected to the input end of the dimension transformation module, and the output end of the dimension transformation module is used to output the extracted image features; the basic convolution module is used to perform preliminary feature extraction on the image to be processed to process and obtain preliminary feature information, and the feature extraction module is used to perform deep feature extraction and fusion on the preliminary feature information to obtain a feature matrix; the dimension transformation module is used to perform dimensionality reduction processing on the feature matrix to obtain a one-dimensional feature vector; the classification probability output module is used to perform probability extraction on the one-dimensional feature vector to obtain the probability corresponding to the feature vector; wherein the feature extraction module includes multiple serially connected feature extraction submodules. When the network model of this embodiment is used to extract image feature information, multiple feature submodules can perform multi-scale spatial feature extraction on the image, so that the extracted feature information includes multiple attribute information, which improves the accuracy of feature extraction and improves the speed of feature information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the overall structure of the network model of an embodiment of the present application;

[0047] Figure 2This is a schematic diagram of the overall structure of the feature extraction submodule of an embodiment of the present application;

[0048] Figure 3 A schematic diagram of the specific structure of the feature extraction submodule of an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the overall structure of the PSA module of an embodiment of the present application;

[0050] Figure 5 This is a schematic diagram of the overall specific structure of the PSA module of an embodiment of the present application;

[0051] Figure 6 Schematic diagram of the structure of the feature extraction device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0053] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0054] The serial numbers assigned to the components in this article, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning.

[0055] A deep learning network model with an attention mechanism proposed in the present application can be applied to multiple fields and used to extract feature information, so that the extracted feature information includes multiple attribute information, which improves the accuracy of feature extraction and the speed of feature information extraction. For example, the network model of the present application is used to extract the driver's state. First, a large number of driver monitoring pictures of multiple concerned feature information are extracted to form a training set. After the model is trained with the training set, the extraction accuracy reaches the requirements. The trained network model can be deployed with the equipment to extract features from the monitored pictures in real time, and then identify whether the driver's current state meets the requirements for safe driving. When it is detected that the current state does not meet the requirements for safe driving, timely external intervention is carried out to avoid dangerous driving. For example, when the driver is identified to have the state information of lowering his head and squinting his eyes, it is determined that the driver may be drowsy or sleepy, and an alarm prompt sound is immediately issued to remind the driver to drive safely.

[0056] Embodiment 1:

[0057] Please refer to Figure 1 , this embodiment provides a deep learning network model with an attention mechanism, which includes: a basic convolution module 1, a feature extraction module 2, a dimension transformation module 3, and a classification probability output module 4.

[0058] Among them, the input end of the basic convolution module 1 is used to input the image to be processed, the output end of the basic convolution module 1 is connected to the input end of the feature extraction module 2, the output end of the feature extraction module 2 is connected to the input end of the dimension transformation module 3, and the output end of the dimension transformation module 3 is connected to the input end of the classification probability output module 4. The basic convolution module 1 of this embodiment is a basic convolution module conventionally selected by a person skilled in the art, and the basic convolution module 1 is used to perform preliminary feature extraction processing on the image to be processed to obtain preliminary feature information. The feature extraction module 2 is used to perform deep feature extraction and fusion on the preliminary feature information to obtain a feature matrix; the dimension transformation module 3 is used to perform dimensionality reduction processing on the feature matrix, and transform it into a feature vector matching the output dimension. The output dimension refers to the state quantity, such as 3 states, each state is two classifications, and the output dimension is 3*2; the classification probability output module 4 is used to perform probability calculation on the feature vector to obtain the probability corresponding to each state classification, for example, state 1: whether to close the eyes is divided into closed eyes and not closed eyes, state 2: whether to smoke is divided into smoking, not smoking, etc. Among them, the dimensionality transformation module 3 of this embodiment includes a reshape function (the reshape function is a function in MATLAB that transforms a specified matrix into a matrix of a specific dimension, and the number of elements in the matrix remains unchanged. The function can readjust the number of rows, columns, and dimensions of the matrix), and the classification probability output module 4 includes a softmax function.

[0059] The network model feature extraction module 2 of this embodiment includes multiple serially connected feature extraction sub-modules 21. Generally, 4-6 serially connected feature extraction sub-modules 21 can be set. In some cases, the number of serially connected feature extraction sub-modules 21 can also be increased according to actual conditions. The structure of each feature extraction sub-module 21 is the same. Multiple feature extraction sub-modules 21 can make the extracted feature information more accurate.

[0060] The feature extraction submodule of this embodiment adds an attention mechanism, which can extract multi-dimensional feature information of the image at one time, while improving the accuracy of feature information extraction. Figure 2 , which includes a first convolution module, a second convolution module, a first channel, a second channel, a third channel, a fourth channel, and a first adder.

[0061] The output end of the first convolution module is respectively connected to the input end of the first channel, the second channel, the third channel, and the fourth channel, and the output end of the first channel, the second channel, the third channel, and the fourth channel are connected to the input end of the second convolution module after passing through the first adder. The first channel includes a first attention mechanism module, a first AG module, and a first multiplier. The output end of the first attention mechanism module is also connected to the input end of the first AG module. The output end of the first AG module and the output end of the first attention mechanism module are connected to the first adder after passing through the first multiplier.

[0062] The second channel includes a third convolution module, a second attention mechanism module, a second AG module and a second multiplier; the input end of the third convolution module is connected to the output end of the first convolution module, and the output end of the third convolution module is connected to the input end of the second attention mechanism module; the output end of the second attention mechanism module is connected to the input end of the second AG module, and the output end of the second attention mechanism module and the output end of the second AG module are also connected to the first adder after passing through the second multiplier.

[0063] The third channel includes a fourth convolution module, a fifth convolution module, a third attention mechanism module, a third AG module and a third multiplier; the input end of the fourth convolution module is connected to the output end of the first convolution module, and the output end of the fourth convolution module is connected to the input end of the fifth convolution module; the output end of the fifth convolution module is connected to the input end of the third attention mechanism module, the output end of the third attention mechanism module is connected to the input end of the third AG module, and the output end of the third AG module and the output end of the third attention mechanism module are also connected to the first adder after passing through the third multiplier.

[0064] The fourth channel includes a sixth convolution module, a seventh convolution module, an eighth convolution module, a fourth attention mechanism module, a fourth AG module and a fourth multiplier; the input end of the sixth convolution module is connected to the output end of the first convolution module, the output end of the sixth convolution module is connected to the input end of the seventh convolution module, and the output end of the seventh convolution module is connected to the input end of the eighth convolution module; the output end of the eighth convolution module is connected to the input end of the fourth attention mechanism module, the output end of the fourth attention mechanism module is connected to the input end of the fourth AG module, and the output end of the fourth AG module and the output end of the fourth attention mechanism module are connected to the first adder after passing through the fourth multiplier.

[0065] In this embodiment, the first convolution module is used to perform dimension processing on the input feature information to obtain feature information of the target dimension; the first channel, the second channel, the third channel, and the fourth channel are respectively used to extract features from the feature information of the target dimension to obtain feature streams of different scales.

[0066] Among them, in the first channel, the first attention mechanism module is used to extract the feature information of the target dimension and add the attention mechanism to obtain the first feature stream; the first AG module is used to assign the corresponding weight to the first feature stream output by the first attention mechanism module to obtain the first weight feature stream; finally, the first feature stream and the first weight feature stream are multiplied to obtain the first scale feature stream. In the second channel, the third convolution module is used to extract the feature information of the target dimension to obtain coarse-grained feature information; the second attention mechanism module is used to further extract the coarse-grained feature information to obtain a fine-grained second feature stream; the second AG module is used to assign the corresponding weight to the second feature stream to obtain the second weight feature stream; finally, the second feature stream and the second weight feature stream are multiplied to obtain the second scale feature stream. In the third channel, the fourth convolution module and the fifth convolution module are respectively used to extract the feature information of the target dimension in turn, and then the extracted features are further extracted through the third attention mechanism module to obtain the third feature stream; the third AG module is used to assign the corresponding weight to the third feature to obtain the third weight feature stream; finally, the third feature stream and the third weight feature stream are multiplied to obtain the third scale feature stream. In the fourth channel, the sixth convolution module, the seventh convolution module and the eighth convolution module are respectively used to extract the feature information of the target dimension in turn, and then the extracted features are further extracted through the fourth attention mechanism module to obtain the fourth feature stream; the fourth AG module is used to assign corresponding weights to the fourth features to obtain the fourth weight feature stream; finally, the fourth feature stream and the fourth weight feature stream are multiplied to obtain the fourth scale feature stream. Finally, the first scale feature stream, the second scale feature stream, the third scale feature stream and the fourth scale feature stream are added through the first adder to obtain feature streams of different scales. The second convolution module is used to adjust the dimensions of the output feature streams of different scales so that their dimensions are the same.

[0067] In one embodiment, the deep learning network model also includes a residual module and a second adder, the output end of the second convolution module is connected to the second adder, and the first convolution module is also connected to the second adder after passing through the residual module; the input feature information is residually processed by the residual module and then added to the feature stream of the same dimension output by the second convolution module through the second adder and then output.

[0068] The network model of this embodiment integrates channel multi-scale feature extraction with spatial multi-scale feature extraction, improves on the basis of the osnet network structure, adds the EPSA module of the attention mechanism, and the overall implementation process of data processing is as follows: Figure 1 , where the main unit structure is shown in Figure 2In this embodiment, the first attention mechanism module, the second attention mechanism module, the third attention mechanism module, and the fourth attention mechanism module are all EPSA modules (i.e., Efficient Pyramid Split Attention, attention segmentation module, Figure 3 The first AG module, the second AG module, the third AG module, and the fourth AG module can be called unified aggregation gates, and the implementation inside is to use global average pooling plus full connection, which is similar to the SE module, but not as complicated as the SE module. The final result is the weight value of each information flow. The osnet network itself uses deep separable convolution, i.e., lite3*3 convolution, with few model parameters and fast calculation. This embodiment adds a multi-scale channel attention mechanism without increasing too much calculation to improve the model's ability to extract key features of images.

[0069] For details, please refer to Figure 3 In this embodiment, the first convolution module and the second convolution module are both 1×1 convolution modules; the third convolution module, the fourth convolution module, the fifth convolution module, the sixth convolution module, the seventh convolution module, and the eighth convolution module are all Lite3×3 convolution modules.

[0070] In another embodiment, the network model can also be changed Figure 3 The relative positions and numbers of the EPSA modules and Lite3*3 modules in the present embodiment select a structure that performs channel feature map weight calibration after deep feature extraction, i.e., places the EPSA module below Lite3*3. In order not to increase the calculation parameters, only one EPSA module is used for each channel in the present embodiment.

[0071] Among them, the first attention mechanism module, the second attention mechanism module, the third attention mechanism module, and the fourth attention mechanism module all include the PSA module. The PSA module is the core module of the EPSA module. First, the feature map is divided into 4 parts in the channel dimension, and each channel feature is subjected to 1 to 4 3*3 convolutions, which will obtain convolution results of different receptive fields. Here, in order to reduce parameters, the lite3*3 convolution structure in osnet is used. The four parts of the channel features finally obtained are processed by the SE module respectively, that is, the channel attention weights are extracted, and finally the 4 parts of the values ​​are softmaxed and concat (splicing) with the feature stream obtained from the previous channel, and finally the four parts of the channel feature stream are multiplied to obtain the feature map after the multi-scale feature information attention weighting.

[0072] This embodiment Figure 3In the example, Input is the feature information passed by the upper network. After being transformed into a suitable dimension by 1*1 convolution, it is sent to four channels respectively. The first channel extracts features by an EPSA model. The second channel first extracts features through a Lite3*3 module and then connects to EPSA for deep extraction. The third channel first passes through two Lite3*3 modules in sequence and then connects to EPSA. The fourth channel is three Lite3*3 modules and then EPSA. The four channels have different sizes of receptive fields due to the different numbers of Lite3*3 modules they pass through. Then, the attention mechanism is added to EPSA. This module can effectively extract more fine-grained multi-scale spatial information and recalibrate the feature information. The four channels obtain feature streams of different scales respectively. In order to learn full-scale features, these feature streams are combined in a dynamic way, that is, these feature streams are assigned different weights by a unified aggregation gate AG, which is shared by the four channels here. After that, the information of the four channels is added and the dimension is adjusted by 1*1 convolution. In addition, the input information is directly connected to the processed feature information through a shortcut, i.e., the residual design. This shotcut connection allows the smaller-scale features learned in the current layer to be effectively preserved in the next layer, so that the final features can capture the entire spatial scale range and effectively reduce the gradient vanishing and gradient exploding problems.

[0073] Among them, Figure 4 The PSA module of this embodiment includes a fifth channel, a sixth channel, a seventh channel, an eighth channel, a first SE module, a second SE module, a third SE module, and a fourth SE module.

[0074] Specifically, the fifth channel includes a ninth convolution module, the input end of the ninth convolution module is used to input the feature information after the split, and the output end thereof is connected to the input end of the first SE module. The sixth channel includes a tenth convolution module and an eleventh convolution module, the input end of the tenth convolution module is used to input the feature information after the split, and the output end thereof is connected to the input end of the eleventh convolution module, and the output end of the eleventh convolution module is connected to the input end of the second SE module. The seventh channel includes a twelfth convolution module, a thirteenth convolution module, and a fourteenth convolution module; the input end of the twelfth convolution module is used to input the feature information after splitting, the output end of the twelfth convolution module is connected to the input end of the thirteenth convolution module, the output end of the thirteenth convolution module is connected to the input end of the fourteenth convolution module, and the output end of the fourteenth convolution module is connected to the input end of the third SE module; the eighth channel includes a fifteenth convolution module, a sixteenth convolution module, a seventeenth convolution module, and an eighteenth convolution module; the input end of the fifteenth convolution module is used to input the feature information after splitting, the output end of the fifteenth convolution module is connected to the input end of the sixteenth convolution module, the output end of the sixteenth convolution module is connected to the input end of the seventeenth convolution module, the output end of the seventeenth convolution module is connected to the input end of the eighteenth convolution module, and the output end of the eighteenth convolution module is connected to the input end of the fourth SE module.

[0075] The output end of the ninth convolution module, the output end of the eleventh convolution module, the output end of the fourteenth convolution module, and the output end of the eighteenth convolution module are connected in sequence, and feature streams of different dimensions (which can also be understood as feature information or feature matrices) are output respectively, and these feature streams of different dimensions are combined (which can also be understood as splicing) to obtain multi-scale feature information. The output end of the first SE module, the output end of the second SE module, the output end of the third SE module, and the output end of the fourth SE module are connected in sequence, and the four SE modules are used to assign corresponding weights to the feature streams output by the four channels, and then the spliced ​​feature stream is multiplied by the weighted feature stream to obtain multi-scale feature information.

[0076] In one embodiment, the network model also includes a fifth multiplier, and the fifth channel, the sixth channel, the seventh channel, and the eighth channel are respectively used to extract features from the input split feature information to obtain feature streams of different dimensions. The first SE module, the second SE module, the third SE module, and the fourth SE module are respectively used to assign weights to the feature streams output by the fifth channel, the sixth channel, the seventh channel, and the eighth channel. The feature streams of different dimensions output by the fifth channel, the sixth channel, the seventh channel, and the eighth channel are spliced, and the spliced ​​feature streams and the spliced ​​weights are multiplied by the fifth multiplier to complete the recalibration of the feature streams and obtain multi-scale feature information.

[0077] Please refer to Figure 5 In this embodiment, the ninth convolution module, the tenth convolution module, the eleventh convolution module, the twelfth convolution module, the thirteenth convolution module, the fourteenth convolution module, the fifteenth convolution module, the sixteenth convolution module, the seventeenth convolution module, and the eighteenth convolution module are all Lite3×3 convolution modules, and the first SE module, the second SE module, the third SE module, and the fourth SE module are abbreviated as SE modules for convenience, and their structures are the same. In this embodiment, Figure 2 The aggregation gate AG in the module only performs feature weighting in spatial dimension, while the EPSA module recalibrates features not only in spatial dimension but also in channel dimension. After obtaining the input features, the module divides the features into 4 groups in channel dimension. The first group enters a Lite3*3 module for feature extraction, the second group enters 2 Lite3*3 modules for extraction, and so on. The use of different numbers of Lite3*3 will obtain featuremaps of different scales, that is, different in spatial dimension, and the 4 groups are composed of different channels, that is, different in channel dimension. The 4 groups of feature streams obtained after the above processing are processed by SE module and softmax function respectively to obtain the attention weights after the new multi-scale channel interaction, and then the 4 groups of feature streams are spliced ​​in channel dimension and multiplied by the spliced ​​weights to recalibrate the features. The feature map obtained after such processing has richer multi-scale information representation capabilities.

[0078] Embodiment 2:

[0079] A method for extracting features from image information includes: inputting the image information into a deep learning network model as provided in Example 1 to obtain a plurality of feature vectors and a probability corresponding to each feature vector. The graphic features extracted by the method of this embodiment are more accurate and include more feature dimensions, which can reflect more detailed information.

[0080] Embodiment three:

[0081] A feature extraction device, such as Figure 6 , which includes a memory 51 and a processor 52, wherein the memory 51 is used to store a program for constructing the deep learning network model provided in the above-mentioned embodiment 1. It can be understood that the network model is implemented by a program, and the processor 52 is used to extract features of image information by executing the program stored in the memory 51.

[0082] Embodiment 4:

[0083] This embodiment provides a computer-readable storage medium, on which a program is stored. The program can be executed by a processor to implement a method for extracting features from image information.

[0084] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0085] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A lightweight deep learning network model with attention mechanism, characterized by: include: Basic convolution module, feature extraction module, dimension transformation module, classification probability output module; The input end of the basic convolution module is used to input the image to be processed, the output end of the basic convolution module is connected to the input end of the feature extraction module, the output end of the feature extraction module is connected to the input end of the dimension transformation module, and the output end of the dimension transformation module is connected to the input end of the classification probability output module; The basic convolution module is used to perform preliminary feature extraction processing on the processed image to obtain preliminary feature information, and the feature extraction module is used to perform deep feature extraction and fusion on the preliminary feature information to obtain a feature matrix; the dimension transformation module is used to perform dimensionality reduction processing on the feature matrix and transform it into a feature vector that matches the output dimension; the classification probability output module is used to perform probability calculation on the feature vector to obtain the probability corresponding to each state classification; Wherein, the feature extraction module comprises a plurality of feature extraction submodules connected in series; The feature extraction submodule includes a first convolution module, a second convolution module, a first channel, a second channel, a third channel, a fourth channel, and a first adder; The output end of the first convolution module is connected to the input end of the first channel, the second channel, the third channel, and the fourth channel respectively, and the output end of the first channel, the second channel, the third channel, and the fourth channel is connected to the input end of the second convolution module after passing through the first adder; The first channel includes a first attention mechanism module, a first AG module and a first multiplier, the output end of the first attention mechanism module is also connected to the input end of the first AG module, and the output end of the first AG module and the output end of the first attention mechanism module are connected to the first adder after passing through the first multiplier; The second channel includes a third convolution module, a second attention mechanism module, a second AG module and a second multiplier; the input end of the third convolution module is connected to the output end of the first convolution module, and the output end of the third convolution module is connected to the input end of the second attention mechanism module; the output end of the second attention mechanism module is connected to the input end of the second AG module, and the output end of the second attention mechanism module and the output end of the second AG module are also connected to the first adder after passing through the second multiplier; The third channel includes a fourth convolution module, a fifth convolution module, a third attention mechanism module, a third AG module and a third multiplier; the input end of the fourth convolution module is connected to the output end of the first convolution module, and the output end of the fourth convolution module is connected to the input end of the fifth convolution module; the output end of the fifth convolution module is connected to the input end of the third attention mechanism module, the output end of the third attention mechanism module is connected to the input end of the third AG module, and the output end of the third AG module and the output end of the third attention mechanism module are also connected to the first adder after passing through the third multiplier; The fourth channel includes a sixth convolution module, a seventh convolution module, an eighth convolution module, a fourth attention mechanism module, a fourth AG module and a fourth multiplier; the input end of the sixth convolution module is connected to the output end of the first convolution module, the output end of the sixth convolution module is connected to the input end of the seventh convolution module, and the output end of the seventh convolution module is connected to the input end of the eighth convolution module; the output end of the eighth convolution module is connected to the input end of the fourth attention mechanism module, the output end of the fourth attention mechanism module is connected to the input end of the fourth AG module, and the output end of the fourth AG module and the output end of the fourth attention mechanism module are connected to the first adder after passing through the fourth multiplier.

2. The lightweight deep learning network model with attention mechanism as claimed in claim 1, characterized in that: The first convolution module is used to perform dimension processing on the input feature information to obtain feature information of a target dimension; The first channel, the second channel, the third channel, and the fourth channel are respectively used to extract feature information of the target dimension to obtain feature flows of different scales; Among them, in the first channel, the first attention mechanism module is used to extract features from the feature information of the target dimension and add the attention mechanism to obtain a first feature stream; the first AG module is used to assign corresponding weights to the first feature stream output by the first attention mechanism module to obtain a first weight feature stream; finally, the first feature stream and the first weight feature stream are multiplied to obtain a first scale feature stream; In the second channel, the third convolution module is used to extract features from the feature information of the target dimension to obtain coarse-grained feature information; the second attention mechanism module is used to further extract features from the coarse-grained feature information to obtain a fine-grained second feature stream; the second AG module is used to assign corresponding weights to the second feature stream to obtain a second weighted feature stream; finally, the second feature stream and the second weighted feature stream are multiplied to obtain a second scale feature stream; In the third channel, the fourth convolution module and the fifth convolution module are respectively used to extract features of the feature information of the target dimension in sequence, and then the extracted features are further extracted through the third attention mechanism module to obtain a third feature stream; the third AG module is used to assign corresponding weights to the third features to obtain a third weight feature stream; finally, the third feature stream and the third weight feature stream are multiplied to obtain a third scale feature stream; In the fourth channel, the sixth convolution module, the seventh convolution module and the eighth convolution module are respectively used to extract features from the feature information of the target dimension in sequence, and then the extracted features are further extracted through the fourth attention mechanism module to obtain a fourth feature stream; the fourth AG module is used to assign corresponding weights to the fourth features to obtain a fourth weight feature stream; finally, the fourth feature stream and the fourth weight feature stream are multiplied to obtain a fourth scale feature stream; Wherein, the first AG, the second AG, the third AG, and the fourth AG parameters are shared; Adding the first scale feature stream, the second scale feature stream, the third scale feature stream and the fourth scale feature stream through the first adder to obtain the feature streams of different scales; The second convolution module is used to adjust the dimensions of the output feature streams of different scales so that their dimensions are the same.

3. The lightweight deep learning network model with attention mechanism as claimed in claim 1, characterized in that: Also includes a residual module and a second adder; The output end of the second convolution module is connected to the second adder, and the first convolution module is further connected to the second adder after passing through the residual module; The input feature information is subjected to residual processing by the residual module and then added to the feature stream of the same dimension output by the second convolution module and then output.

4. The lightweight deep learning network model with attention mechanism as claimed in claim 1, characterized in that: The first convolution module and the second convolution module are both 1×1 convolution modules; The third convolution module, the fourth convolution module, the fifth convolution module, the sixth convolution module, the seventh convolution module, and the eighth convolution module are all Lite3×3 convolution modules.

5. The lightweight deep learning network model with attention mechanism as claimed in claim 1, characterized in that: The first attention mechanism module, the second attention mechanism module, the third attention mechanism module, and the fourth attention mechanism module all include a PSA module; The PSA module includes a fifth channel, a sixth channel, a seventh channel, an eighth channel, a first SE module, a second SE module, a third SE module, and a fourth SE module; The fifth channel includes a ninth convolution module, the input end of the ninth convolution module is used to input the split feature information, and the output end is connected to the input end of the first SE module; The sixth channel includes a tenth convolution module and an eleventh convolution module, the input end of the tenth convolution module is used to input the split feature information, and the output end is connected to the input end of the eleventh convolution module, and the output end of the eleventh convolution module is connected to the input end of the second SE module; The seventh channel includes a twelfth convolution module, a thirteenth convolution module, and a fourteenth convolution module; the input end of the twelfth convolution module is used to input the split feature information, the output end of the twelfth convolution module is connected to the input end of the thirteenth convolution module, the output end of the thirteenth convolution module is connected to the input end of the fourteenth convolution module, and the output end of the fourteenth convolution module is connected to the input end of the third SE module; The eighth channel includes a fifteenth convolution module, a sixteenth convolution module, a seventeenth convolution module, and an eighteenth convolution module; the input end of the fifteenth convolution module is used to input the split feature information, the output end of the fifteenth convolution module is connected to the input end of the sixteenth convolution module, the output end of the sixteenth convolution module is connected to the input end of the seventeenth convolution module, the output end of the seventeenth convolution module is connected to the input end of the eighteenth convolution module, and the output end of the eighteenth convolution module is connected to the input end of the fourth SE module; The output end of the ninth convolution module, the output end of the eleventh convolution module, the output end of the fourteenth convolution module, and the output end of the eighteenth convolution module are connected in sequence; The output end of the first SE module, the output end of the second SE module, the output end of the third SE module and the output end of the fourth SE module are connected in sequence.

6. The lightweight deep learning network model with attention mechanism as claimed in claim 5, characterized in that: A fifth multiplier is also included; The fifth channel, the sixth channel, the seventh channel, and the eighth channel are respectively used to extract features from the input split feature information to obtain feature streams of different dimensions; The first SE module, the second SE module, the third SE module, and the fourth SE module are used to calculate weights for the feature streams output by the fifth channel, the sixth channel, the seventh channel, and the eighth channel, respectively; The feature streams of different dimensions output by the fifth channel, the sixth channel, the seventh channel, and the eighth channel are spliced, and the spliced ​​feature stream and the spliced ​​weight vector are multiplied by the fifth multiplier to complete the recalibration of the feature stream and obtain multi-scale feature information.

7. A method for extracting features from image information, characterized in that: include: The image information is input into the deep learning network model as described in any one of claims 1 to 6 to obtain multiple feature vectors and the probability corresponding to each feature vector.

8. A feature extraction device, characterized in that: include: A memory for storing a program for constructing the deep learning network model according to any one of claims 1 to 6; The processor is used to extract features of the image information by executing the program stored in the memory.

9. A computer-readable storage medium, characterized in that: The medium stores a program, which can be executed by a processor to implement the feature extraction method of image information as claimed in claim 7.

Images