An image recognition method and device based on efficient multi-type convolution aggregation convolution

By integrating multiple convolution operations and complementary self-attention aggregation mechanisms, the efficiency and adaptability problems of traditional CNNs in feature extraction and fusion are solved, and more efficient feature aggregation and more reliable image recognition performance are achieved.

CN117746209BActive Publication Date: 2024-11-22SHANDONG INSPUR ULTRA HD INTELLIGENT TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202311707124.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-13
Publication Date
2024-11-22
Estimated Expiration
2043-12-13

AI Technical Summary

Technical Problem

Traditional CNNs have problems such as reduced learning efficiency, loss of small sample features and unreliable feature expression information in terms of feature extraction and fusion. The fixed-shaped receptive field cannot capture global information, which limits its adaptability to different data features.

Method used

The image recognition method based on efficient multi-type convolution aggregation is adopted to integrate various convolution operations such as standard convolution, expansion convolution, ODConv, dynamic serpentine convolution, and dynamically select and fusion features through complementary self-attention aggregation mechanism. Achieve comprehensive feature exploration, extraction and representation.

Benefits of technology

It improves the performance of image processing, captures richer spatial global information clues, and realizes efficient latent feature aggregation and more reliable feature expression.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117746209B_ABST
    Figure CN117746209B_ABST
Patent Text Reader

Abstract

The present invention provides an image recognition method and device based on efficient multi-type convolution aggregation convolution, belonging to the field of computer vision technology. It is achieved through the following technical scheme: collecting images to be recognized, and pre-processing the images, dividing them into test sets, verification sets and training sets; constructing an image recognition model based on efficient multi-type convolution aggregation convolution, the model includes a feature aggregation module and a CSAAM information extraction and fusion module; training the image recognition model based on efficient multi-type convolution aggregation convolution through the training set, verifying the model performance through the verification set, and obtaining the model with the best performance; inputting the test set data into the trained image recognition model based on efficient multi-type convolution aggregation convolution, and outputting the image recognition result. The present invention integrates multiple types of convolution operations into a single network technology, realizes comprehensive feature extraction and representation, and proposes a complementary self-attention aggregation mechanism based on the parallel strategy to realize efficient potential feature aggregation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image recognition method and device based on efficient multi-type convolution aggregation convolution, belonging to the technical field of computer vision. Background Art

[0002] Currently, CNN has achieved remarkable success in computer vision tasks. Its basic framework is to achieve feature extraction and fusion through stacked convolution, normalization, activation, Add / Cat and other operations. However, with the continuous development of models and the expansion of data types, traditional feature extraction and fusion methods are facing many challenges. The method of stacking standard convolution blocks to increase the width and depth of the network can effectively capture long-range feature dependencies, but it also causes problems such as reduced learning efficiency and loss of small sample features. Standard static convolution often extracts features through a 3x3 or 5x5 square receptive field, which cannot count global information in one convolution and forms a decoupled relationship with the input data. At the same time, the fixed-shape receptive field cannot establish a strong correlation with pure features. On the other hand, standard convolution only focuses on the spatial and channel dimensions, and assumes the same contribution within and between dimensions, resulting in insufficient feature extraction and unreliable feature expression information.

[0003] Based on this, a large number of improved convolution and fusion strategies have emerged. CondConv is the cornerstone of dynamic convolution. It is similar to the word embedding matrix in CBOW. Whenever a one-hot encoded token is brought in, it will be automatically mapped to the embedding matrix to obtain its dense vector representation. In CondConv convolution, FM is used as a token, FM's GlobalAveragePooling is used as a one-hot encoding, the matrix R is used as a word embedding matrix, and the weight vectors of n experts are used as dense vector representations. The final weight acts on the convolution kernel instead of FM, which greatly reduces the amount of calculation. The difference between DyConv and CondConv is that the weight vector is obtained through two fully connected layers. DyConv and CondConv couple the convolution operation with the input data through the action of multiple weighted experts, implicitly increase the network width and improve the model capability. This belongs to kernel-level attention, but this method only focuses on the dimension of the number of convolution kernels, ignores other dimensions of the convolution kernel space, and fails to fully stimulate the potential of dynamic convolution.

[0004] Moreover, traditional CNNs usually rely on a single type of convolution operation, which limits their adaptability to different data features. Although the dynamic convolution method can shallowly adapt to data features, its performance may be affected when faced with subtle local features and complex and changeable global forms. Summary of the invention

[0005] The purpose of the present invention is to provide an image recognition method and device based on efficient multi-type convolution aggregation convolution, which realizes comprehensive feature exploration, extraction and representation, and realizes efficient potential feature aggregation, thereby improving the performance of image processing.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] Collect the images to be identified, pre-process them, and divide them into test set, validation set and training set;

[0008] Construct an image recognition model based on efficient multi-type convolution aggregation convolution, which includes a feature aggregation module and a CSAAM information extraction and fusion module;

[0009] Use the training set to train the image recognition model based on efficient multi-type convolution aggregation convolution, and use the validation set to verify the model performance to obtain the model with the best performance;

[0010] Input the test set data into the trained image recognition model based on efficient multi-type convolution aggregation convolution, and output the image recognition results.

[0011] Preferably, the feature aggregation module aggregates five types of convolution operations, including D-Conv, C-CDC, Conv, DSC-Conv and OD-Conv;

[0012] The two sub-operators of C-CDC split horizontally, vertically and diagonally are decoupled from the standard convolution; the DSC-Conv is split separately from three perspectives, introducing two perspectives expanded along the x and y directions;

[0013] The five output features of D-Conv, C-CDC, Conv and OD-Conv operations are processed by the BS module and the two output features of DSC-Conv operation are processed by the GS module and merged as the output of the feature aggregation module and input into the CSAAM information extraction and fusion module.

[0014] Preferably, the BS module is a combination of BN batch normalization and SiLU activation function, and the GS module is a combination of GN group normalization and SiLU activation function.

[0015] Preferably, the CSAAM information extraction and fusion module performs global information extraction and fusion of three spatial perspectives, including a discarding merging module and a dynamic weighting module.

[0016] Preferably, the discarding and merging module includes three parallel Self-Attention modules, and the specific processing process is as follows:

[0017] The outputs of the feature aggregation module are input into three Self-Attention modules respectively. The outputs of the three Self-Attention modules are randomly discarded, and only one-third of the channels are retained. They are then merged by channel and input into the dynamic weighting module.

[0018] Preferably, the specific structure of the Self-Attention module is as follows:

[0019] The outputs of the feature aggregation module are respectively input into the first, second and third parallel convolutional layers. The output results of the first and second convolutional layers are matrix multiplied and then processed by the Softmax function and matrix multiplied with the output results of the third convolutional layer. The output results are then randomly discarded, retaining only one-third of the number of channels.

[0020] Preferably, the specific processing process of the dynamic weighting module is as follows:

[0021] The output of the discard merging module is processed by a 1×1 convolutional layer and then expanded through the BS module to make it consistent with the output channel of the feature aggregation module. The output features after the expanded channel are then merged with the output features of the feature aggregation module and brought into the SE module to solve the dynamic channel weights. The output features after the expanded channel are dynamically weighted according to the weights as output to achieve complementary fusion of the three spatial perspective features.

[0022] The advantages of the present invention are that: the present invention integrates various types of convolution operations into a single network technology. By integrating various convolution operations such as standard convolution, dilated convolution, ODConv, dynamic snake convolution, C-CDC, etc., EMC3 realizes comprehensive feature exploration, extraction and representation.

[0023] Based on the parallel strategy, a complementary self-attention aggregation mechanism is proposed to dynamically select and fuse the most informative features from different convolution operations to capture richer spatial global information clues. Based on SENet, the captured clues are dynamically selected and fused, and CSAAM achieves efficient potential feature aggregation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0025] Figure 1 It is a schematic diagram of the process structure of the present invention.

[0026] Figure 2 This is a schematic diagram of the network structure of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Example 1

[0029] An image recognition method based on efficient multi-type convolution aggregation convolution is implemented through the following technical solutions:

[0030] S1: Collect the face images to be recognized, pre-process the images, and divide them into test set, validation set and training set;

[0031] S2: Construct an image recognition model based on efficient multi-type convolution aggregation convolution, the model includes a feature aggregation module and a CSAAM information extraction and fusion module; the feature aggregation module aggregates five types of convolution operations, including D-Conv, C-CDC, Conv, DSC-Conv and OD-Conv;

[0032] The two sub-operators of C-CDC split horizontally, vertically and diagonally are decoupled from the standard convolution; the DSC-Conv is split separately from three perspectives, introducing two perspectives expanded along the x and y directions;

[0033] The five output features of D-Conv, C-CDC, Conv and OD-Conv are processed by the BS module and the two output features of DSC-Conv are processed by the GS module and merged as the output of the feature aggregation module, and input into the CSAAM information extraction fusion module. The BS module is a combination of BN batch normalization and SiLU activation function, and the GS module is a combination of GN group normalization and SiLU activation function.

[0034] The CSAAM information extraction and fusion module performs global information extraction and fusion of three spatial perspectives, including a discarding merging module and a dynamic weighting module.

[0035] Optionally, the discarding and merging module includes three parallel Self-Attention modules, and the specific processing process is as follows:

[0036] The outputs of the feature aggregation module are input into three Self-Attention modules respectively. The outputs of the three Self-Attention modules are randomly discarded, and only one-third of the channels are retained. They are then merged by channel and input into the dynamic weighting module.

[0037] Optionally, the specific structure of the Self-Attention module is as follows:

[0038] The outputs of the feature aggregation module are respectively input into the first, second and third parallel convolutional layers. The output results of the first and second convolutional layers are matrix multiplied and then processed by the Softmax function and matrix multiplied with the output results of the third convolutional layer. The output results are then randomly discarded, retaining only one-third of the number of channels.

[0039] Optionally, the specific processing process of the dynamic weighting module is as follows:

[0040] The output of the discard merging module is processed by a 1×1 convolutional layer and then expanded through the BS module to make it consistent with the output channel of the feature aggregation module. The output features after the expanded channel are then merged with the output features of the feature aggregation module and brought into the SE module to solve the dynamic channel weights. The output features after the expanded channel are dynamically weighted according to the weights as output to achieve complementary fusion of the three spatial perspective features.

[0041] S3: Train the image recognition model based on efficient multi-type convolution aggregation convolution through the training set, verify the model performance through the validation set, and obtain the model with the best performance;

[0042] S4: Input the test set data into the trained image recognition model based on efficient multi-type convolution aggregation convolution, and output the image recognition results.

[0043] Example 2

[0044] The disclosed embodiment also provides an image recognition device based on efficient multi-type convolution aggregation convolution, including a processor and a memory. Optionally, the device may also include a communication interface and a bus. Among them, the processor, the communication interface, and the memory can communicate with each other through the bus. The communication interface can be used for information transmission. The processor can call the logic instructions in the memory to execute the image recognition method based on efficient multi-type convolution aggregation convolution of the above embodiment.

[0045] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0046] The memory, as a computer-readable storage medium, can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes the function application and data processing by running the program instructions / modules stored in the memory, that is, the image recognition method based on efficient multi-type convolution aggregation convolution in the above embodiment is implemented.

[0047] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory.

[0048] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned image recognition method based on efficient multi-type convolution aggregation convolution.

[0049] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0050] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0051] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image recognition method based on efficient multi-type convolution aggregation convolution, characterized in that: include: Collect the images to be identified, pre-process them, and divide them into test set, validation set and training set; Construct an image recognition model based on efficient multi-type convolution aggregation convolution, which includes a feature aggregation module and a CSAAM information extraction and fusion module; Use the training set to train the image recognition model based on efficient multi-type convolution aggregation convolution, and use the validation set to verify the model performance to obtain the model with the best performance; Input the test set data into the trained image recognition model based on efficient multi-type convolution aggregation convolution, and output the image recognition results; The feature aggregation module aggregates five types of convolution operations, including D-Conv, C-CDC, Conv, DSC-Conv and OD-Conv; The two sub-operators of C-CDC split horizontally, vertically and diagonally are decoupled from the standard convolution; the DSC-Conv is split separately from three perspectives, introducing two perspectives expanded along the x and y directions; The five output features of D-Conv, C-CDC, Conv and OD-Conv are processed by the BS module and the two output features of DSC-Conv are processed by the GS module, and then merged as the output of the feature aggregation module and input into the CSAAM information extraction and fusion module; The CSAAM information extraction and fusion module performs global information extraction and fusion of three spatial perspectives, including a discarding merging module and a dynamic weighting module; The discarding and merging module includes three parallel Self-Attention modules; The specific processing process of the dynamic weighting module is as follows: The output of the discard merging module is processed by a 1×1 convolutional layer and then expanded through the BS module to make it consistent with the output channel of the feature aggregation module. The output features after the expanded channel are then merged with the output features of the feature aggregation module and brought into the SE module to solve the dynamic channel weights. The output features after the expanded channel are dynamically weighted according to the weights as output to achieve complementary fusion of the three spatial perspective features.

2. The image recognition method based on efficient multi-type convolution aggregation convolution according to claim 1 is characterized in that: The BS module is a combination of BN batch normalization and SiLU activation function, and the GS module is a combination of GN group normalization and SiLU activation function.

3. The image recognition method based on efficient multi-type convolution aggregation convolution according to claim 1 is characterized in that: The specific processing process of the discarding and merging module is as follows: The outputs of the feature aggregation module are input into three Self-Attention modules respectively. The outputs of the three Self-Attention modules are randomly discarded, and only one-third of the channels are retained. They are then merged by channel and input into the dynamic weighting module.

4. The image recognition method based on efficient multi-type convolution aggregation convolution according to claim 3 is characterized in that: The specific structure of the Self-Attention module is as follows: The outputs of the feature aggregation module are respectively input into the first, second and third parallel convolutional layers. The output results of the first and second convolutional layers are matrix multiplied and then processed by the Softmax function and matrix multiplied with the output results of the third convolutional layer. The output results are then randomly discarded, retaining only one-third of the number of channels.

5. An image recognition device based on efficient multi-type convolution aggregation convolution, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the image recognition method based on efficient multi-type convolution aggregation convolution as described in any one of claims 1 to 4 when running the program instructions.

6. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the program is executed by a processor, the method as described in any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • Image classification method based on SE module and self-attention mechanism network

    CN113537243A

  • Remote sensing image recognition method and device, computer equipment and storage medium

    CN116030293A

  • Unmanned aerial vehicle detection method based on multi-cavity convolution and SE attention residual error

    CN116310273A

  • Method for recognizing face using multiple patch combination based on deep neural network with fault tolerance and fluctuation robustness in extreme situation

    US20200250402A1

  • Graph convolutional networks with motif-based attention

    US20200285944A1