Image classification method, device and equipment based on multi-scale attention mechanism

Abstract

Embodiments of the present invention provide an image classification method, apparatus and device based on a multi-scale attention mechanism. The method comprises acquiring a target image, wherein the target image comprises semantic information; inputting the target image into a semantic feature extraction module to obtain image semantic features; and inputting the target image into a saliency feature extraction module to obtain image saliency features; weightedly fusing the image semantic features with the image saliency features to obtain saliency weighted semantic features; inputting the saliency weighted semantic features into a fully connected layer to output the classification result of the target image. In this way, a good hierarchical relationship can be effectively maintained, and multi-scale local information and global information can be integrated, which solves the contradiction that the traditional method cannot maintain a good hierarchical relationship and contextual relationship under reasonable computational complexity conditions, while improving the learning ability of the model.

Description

Technical Field

[0001] The present invention generally relates to the field of image classification, and more specifically, to an image classification method, apparatus and device based on a multi-scale attention mechanism. Background Art

[0002] Image classification is one of the core issues in computer vision and the basis of various visual recognition fields such as target detection, human pose estimation, video classification, and target tracking. Image feature extraction has always been the focus of image classification research and the basis of image classification. Traditional image feature extraction algorithms focus on the extraction of manual features, which are often not accurate enough, resulting in poor generalization and portability of the method. The emergence of convolutional neural networks (CNNs) has promoted the development of this field and gradually become the mainstream algorithm for image classification. Unlike traditional image classification methods, CNN-based classification methods provide a good feature learning method and improve overall performance. However, CNN is mainly designed for local feature extraction and is difficult to capture global contextual dependencies, which makes performance evaluation difficult and less standardized, thereby limiting the evaluation performance of classification tasks. Compared with CNN, Transformer can effectively capture global context information. At the same time, it contains a self-attention mechanism, can adaptively enhance features, and obtain reliable global context information, showing certain advantages in the field of image classification. However, the traditional Transformer architecture still faces challenges in utilizing visual features of different scales during feature learning, lacks local information, and has relatively large computational complexity. It still faces many challenges in image classification tasks. Summary of the invention

[0003] According to an embodiment of the present invention, an image classification scheme based on a multi-scale attention mechanism is provided. This scheme can effectively maintain a good hierarchical relationship, integrate multi-scale local information and global information, solve the contradiction that traditional methods cannot maintain a good hierarchical relationship and contextual relationship under reasonable computational complexity conditions, and improve the learning ability of the model.

[0004] In a first aspect of the present invention, a method for image classification based on a multi-scale attention mechanism is provided. The method comprises:

[0005] Acquire a target image, wherein the target image includes semantic information;

[0006] Inputting the target image into a semantic feature extraction module to obtain image semantic features; and inputting the target image into a salient feature extraction module to obtain image salient features;

[0007] The image semantic features are weightedly fused with the image saliency features to obtain saliency weighted semantic features;

[0008] The saliency weighted semantic features are input into a fully connected layer, and the classification result of the target image is output.

[0009] Furthermore, the semantic feature extraction module includes four consecutive stages; the first stage includes a Patch embedding layer and a first DM-former module in sequence, wherein the first stage is used to receive a target image; the second stage includes a first Patch merging layer and a second DM-former module in sequence; the third stage includes a second Patch merging layer and a third DM-former module in sequence; the fourth stage includes a third Patch merging layer and a fourth DM-former module in sequence; the output of the fourth stage is used as an image semantic feature.

[0010] Further, the first DM-former module, the second DM-former module, the third DM-former module and the fourth DM-former module have the same structure, and sequentially include a multi-level residual structure and a residual block;

[0011] The multi-level residual structure includes several improved residual structures, and the output results of each improved residual structure are added and fused, and the output result is used as the output result of the multi-level residual structure; each improved residual structure includes a deep convolution layer, a first LN layer and a multi-head self-attention module MHA in sequence;

[0012] The deep convolution layer obtains input features, encodes the relative position information of the input features, adds and fuses the obtained encoding results with the input features, and outputs the relative position information encoding; the relative position information encoding is input into the first LN layer, the output result of the first LN layer is input into the multi-head self-attention module MHA, the output result of the multi-head self-attention module MHA is added and fused with the relative position information encoding to obtain the output result of the improved residual structure;

[0013] The residual block includes a second LN layer and a multi-layer perceptron MLP in sequence; the second LN layer obtains the output result of the multi-level residual structure, and outputs the residual result through the multi-layer perceptron MLP, and adds and fuses the residual result with the output result of the multi-level residual structure to output as a semantic feature.

[0014] Furthermore, in the multi-level residual structure, different improved residual structures have different scales of deep convolutional layers.

[0015] Furthermore, the step of inputting the target image into a semantic feature extraction module to obtain image semantic features includes:

[0016] In the first stage, the target image is divided into blocks and the blocks are embedded using the Patch embedding layer to obtain the image block feature matrix. The image block feature matrix is ​​then input into the first DM-former module to output the first-level semantic features.

[0017] In the second stage, the first-level semantic features are input into the first Patch merging layer, and the output result of the first Patch merging layer is input into the second DM-former module to output the second-level semantic features;

[0018] In the third stage, the second-level semantic features are input into the second Patch merging layer, and the output result of the second Patch merging layer is input into the third DM-former module to output the third-level semantic features;

[0019] In the fourth stage, the three-level semantic features are input into the third Patch merging layer, and the output result of the third Patch merging layer is input into the fourth DM-former module to output the image semantic features.

[0020] Furthermore, the salient feature extraction module includes a saliency map extraction module and a ResNet50 model in sequence;

[0021] The target image is input into the saliency map extraction module, and a saliency map is output; the saliency map is input into the ResNet50 model, and image saliency features are output.

[0022] Furthermore, the weighted fusion of the image semantic features and the image saliency features to obtain saliency weighted semantic features includes:

[0023] Multiplying the image semantic feature with the image saliency feature to obtain a multiplied fusion feature;

[0024] The multiplied fusion feature is added to the image semantic feature to obtain a saliency weighted semantic feature.

[0025] In a second aspect of the present invention, an image classification device based on a multi-scale attention mechanism is provided. The device comprises:

[0026] An acquisition module, used for acquiring a target image, wherein the target image includes semantic information;

[0027] A feature extraction module, wherein the feature extraction module includes an image semantic feature extraction module and an image saliency feature extraction module; the image semantic feature extraction module is used to input the target image into the semantic feature extraction module to obtain image semantic features; the image saliency feature extraction module is used to input the target image into the saliency feature extraction module to obtain image saliency features;

[0028] A feature fusion module, used for weighted fusion of the image semantic features and image saliency features to obtain saliency weighted semantic features;

[0029] The output module is used to input the saliency weighted semantic features into the fully connected layer and output the classification result of the target image.

[0030] In a third aspect of the present invention, an electronic device is provided. The electronic device has at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method of the first aspect of the present invention.

[0031] In a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the first aspect of the present invention.

[0032] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0034] Figure 1 A flowchart of an image classification method based on a multi-scale attention mechanism according to an embodiment of the present invention is shown;

[0035] Figure 2 A flowchart of semantic feature extraction according to an embodiment of the present invention is shown;

[0036] Figure 3 A flow chart of a weighted fusion module according to an embodiment of the present invention is shown;

[0037] Figure 4 A flowchart of an image classification method based on a multi-scale attention mechanism according to an embodiment of the present invention is shown;

[0038] Figure 5 The ResNet50 architecture diagram of an embodiment of the present invention is shown;

[0039] Figure 6 A flow chart of a DM-former module of a three-level residual structure according to an embodiment of the present invention is shown;

[0040] Figure 7 A block diagram of an image classification device based on a multi-scale attention mechanism according to an embodiment of the present invention is shown;

[0041] Figure 8 A block diagram of an exemplary electronic device capable of implementing embodiments of the present invention is shown;

[0042] Among them, 800 is an electronic device, 801 is a computing unit, 802 is a ROM, 803 is a RAM, 804 is a bus, 805 is an I / O interface, 806 is an input unit, 807 is an output unit, 808 is a storage unit, and 809 is a communication unit. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0045] In the present invention, the acquired target image is input into a feature extraction module and a salient feature extraction module respectively, and then the output results of the above two modules are input into a weighted fusion module to obtain salient weighted semantic features, and finally the salient weighted semantic features are input into a fully connected layer to obtain the classification result of the target image. The method adopts a multiplication and addition fusion method, which can effectively enhance the semantic features contained in the visual salient area under the premise of ensuring all visual information, and can strengthen the key semantic information of the classification target, thereby producing a more accurate image classification result.

[0046] like Figure 1 The flowchart of the image classification method based on the multi-scale attention mechanism of the embodiment of the present invention is shown in FIG. Figure 4The flowchart of the image classification method based on the multi-scale attention mechanism of an embodiment of the present invention is shown.

[0047] The method includes:

[0048] S101. Acquire a target image, where the target image includes semantic information.

[0049] S102, inputting the target image into a semantic feature extraction module M1 to obtain image semantic features; and inputting the target image into a salient feature extraction module M2 to obtain image salient features.

[0050] The semantic feature extraction module M1 described in the first part is as follows:

[0051] Figure 2 A flowchart of semantic feature extraction according to an embodiment of the present invention is shown.

[0052] In this embodiment, the step of inputting the target image into a semantic feature extraction module to obtain image semantic features includes:

[0053] S201. In the first stage, the target image is divided into blocks and the blocks are embedded using the Patch embedding layer to obtain an image block feature matrix; and the image block feature matrix is ​​input into the first DM-former module to output the first-level semantic features.

[0054] S202. In the second stage, the first-level semantic features are input into the first Patch merging layer, and the output result of the first Patch merging layer is input into the second DM-former module to output the second-level semantic features.

[0055] S203. In the third stage, the secondary semantic features are input into the second Patch merging layer, and the output result of the second Patch merging layer is input into the third DM-former module to output the tertiary semantic features.

[0056] S204. In the fourth stage, the three-level semantic features are input into the third Patch merging layer, and the output result of the third Patch merging layer is input into the fourth DM-former module to output the image semantic features.

[0057] Among them, the first DM-former module is repeated N1 times, the second DM-former module is repeated N2 times, the third DM-former module is repeated N3 times, and the fourth DM-former module is repeated N4 times.

[0058] In this embodiment, the semantic feature extraction module includes four consecutive stages; the first stage includes a Patch embedding layer and a first DM-former module in sequence, wherein the first stage is used to receive a target image; the second stage includes a first Patch merging layer and a second DM-former module in sequence; the third stage includes a second Patch merging layer and a third DM-former module in sequence; the fourth stage includes a third Patch merging layer and a fourth DM-former module in sequence; the output of the fourth stage is used as an image semantic feature.

[0059] Specifically, the semantic feature extraction module adopts the pyramid structure of the traditional vision transformer with four stages. For example, the length of the original image is H, the width is W, and the number of channels C is 3, that is, the size of the original image is H*W*3; the size of the first stage Stage1 is 1 / 4 of the original image, that is, the scale of Stage1 is H / 4*W / 4*C, the size of the second stage Stage2 is 1 / 8 of the original image, that is, the scale of Stage2 is H / 8*W / 8*2C, the size of the third stage Stage3 is 1 / 16 of the original image, that is, the scale of Stage3 is H / 16*W / 16*4C, and the size of the fourth stage Stage4 is 1 / 32 of the original image, that is, the scale of Stage4 is H / 32*W / 32*8C.

[0060] The pyramid structure design enables the model to capture rich information at different scales, thus improving the performance and accuracy of the model. Unlike the traditional vision transformer, the emerging merging module is used in the following three stages 2, 3, and 4, which can reduce the surge in spatial resolution caused by the increase in the number of channels and effectively integrate semantic information at different levels.

[0061] In this embodiment, the first DM-former module, the second DM-former module, the third DM-former module and the fourth DM-former module have the same structure, which include a multi-level residual structure and a residual block in sequence.

[0062] The multi-level residual structure includes several improved residual structures, and the output results of each improved residual structure are added and fused, and the output result is used as the output result of the multi-level residual structure; each improved residual structure includes a deep convolution layer DWConv, a first LN layer and a multi-head self-attention module MHA in sequence.

[0063] Among them, the deep convolution layer obtains input features, encodes the relative position information of the input features, adds and fuses the obtained encoding results with the input features, and outputs the relative position information code; the relative position information code is input into the first LN layer, the output result of the first LN layer is input into the multi-head self-attention module MHA, the output result of the multi-head self-attention module MHA is added and fused with the relative position information code to obtain the output result of the improved residual structure.

[0064] The residual block includes a second LN layer and a multi-layer perceptron MLP in sequence; the second LN layer obtains the output result of the multi-level residual structure, and outputs the residual result through the multi-layer perceptron MLP, and adds and fuses the residual result with the output result of the multi-level residual structure to output as semantic information.

[0065] In this embodiment, in the multi-level residual structure, the scales of the depth convolutional layers of different improved residual structures are different.

[0066] Setting deep convolution before the input of the traditional vision Transformer to encode relative position information can effectively suppress local redundancy and achieve sparse coding; using deep convolutions of different scales can obtain attention information of different scales and effectively capture the hierarchical structure and contextual information of the target; multi-scale information shares an LN and MLP layer, which can enhance the interaction between features of different scales, while reducing computational complexity and the difficulty of learning the model.

[0067] In some optional implementations of this embodiment, the multi-level residual structure is a three-level residual structure, and the scales of the deep convolutional layers are 1×1, 3×3, 1, and 5×5, respectively. Figure 6 DM-former module flow chart of the three-level residual structure of the embodiment of the present invention, the specific process is as follows:

[0068] First, three deep convolutional layers obtain the input features of the image block and encode the relative position information of the input features; then, the encoding results of the outputs of the three deep convolutional layers are added and fused with the input features of the image block to obtain the multi-level relative position information encoding of the image block. .

[0069] Secondly, the multi-level relative position information encoding is input into the three first LN layers of the three-level residual structure respectively; then the output results of the three first LN layers are input into three multi-head self-attention modules MHA; then the output results of the three multi-head self-attention modules MHA are added and fused with the multi-level relative position information encoding respectively to obtain the output results of three improved residual structures; then the output results of the three improved residual structures are added and fused to obtain multi-level visual features.

[0070] Finally, the multi-level visual features are input into the second LN layer; then the output result of the second LN layer is input into the multi-layer perceptron MLP to obtain the residual result; then the multi-level visual features and the residual result are added and fused and output as semantic information.

[0071] The salient feature extraction module M2 described in the second part is as follows:

[0072] In this embodiment, the salient feature extraction module includes a saliency map extraction module and a ResNet50 model in sequence.

[0073] The target image is input into the saliency map extraction module, and a saliency map is output; the saliency map is input into the ResNet50 model, and image saliency features are output.

[0074] The extraction of saliency maps can effectively capture visually significant semantic areas, extract multi-scale local hierarchical information, improve the visual representation ability of the model, help the model identify more recognizable key target semantic information, and help improve classification results.

[0075] Specifically, the saliency map extraction module is a combination of a deep convolutional network and an attention mechanism, namely SalDA (DeepConvNet Greets Attention for Visual Saliency). Considering that each feature map may capture different salient areas, SalDA first adaptively aggregates the features of each position by weighting all position features in each independent channel to obtain a spatial attention module. At the same time, in order to capture the interdependence between channels, SalDA also introduces a channel attention module to integrate different characteristics between different channels. SalDA merges these two attention modules into a multi-attention module to further improve the network's saliency map prediction.

[0076] Specifically, Figure 5 The ResNet50 architecture diagram of an embodiment of the present invention is shown, and the network structure is: 1 layer of 7*7 convolutional layer with 64 channels, 1 layer of 3*3 pooling layer, 3 layers of 1*1 convolutional layer with 64 channels + 3*3 convolutional layer with 64 channels + 1*1 convolutional layer with 256 channels, 4 layers of 1*1 convolutional layer with 128 channels + 3*3 convolutional layer with 128 channels + 1*1 convolutional layer with 512 channels, 6 layers of 1*1 convolutional layer with 256 channels + 3*3 convolutional layer with 256 channels + 1*1 convolutional layer with 1024 channels, and 3 layers of 1*1 convolutional layer with 523 channels + 3*3 convolutional layer with 512 channels + 1*1 convolutional layer with 2048 channels.

[0077] S103: weighted fusion of the image semantic features and image saliency features to obtain saliency weighted semantic features.

[0078] Figure 3 A flow chart of a weighted fusion module according to an embodiment of the present invention is shown.

[0079] In this embodiment, the image semantic features and image saliency features are input into a weighted fusion module M3 for weighted fusion to obtain saliency weighted semantic features, including:

[0080] S301, multiplying the image semantic feature with the image saliency feature to obtain a multiplied fusion feature.

[0081] S302: Add the multiplied fusion feature and the image semantic feature to obtain a significant weighted semantic feature.

[0082] The multiplication and addition fusion method can effectively enhance the semantic features contained in the visually significant area while ensuring all visual information, thereby producing more accurate image classification results.

[0083] S104: Input the saliency weighted semantic features into the fully connected layer FC, and output the classification result of the target image.

[0084] In some optional implementations of this embodiment, C=32, 64, 96, N1=2, 4, N2=2, 4, N3=8, 16, N4=2, 4.

[0085] According to an embodiment of the present invention, the semantic feature extraction module can effectively suppress local redundancy through deep convolution, realize sparse coding, and reduce the learning difficulty of the model; the design of the multi-scale convolution layer can realize the interaction between features of different scales and effectively integrate semantic information at different levels; multi-level fusion shares the same residual block learning features, reducing the overall computational complexity; the design of the saliency feature module enhances the feature representation capability and improves the overall performance and accuracy of the image classification results.

[0086] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0087] The above is an introduction to a method embodiment. The following is a further explanation of the scheme of the present invention through a device embodiment having the same inventive concept as the method in the aforementioned embodiment.

[0088] like Figure 7 As shown, the apparatus 700 includes:

[0089] The acquisition module 710 is used to acquire a target image, where the target image includes semantic information.

[0090] The feature extraction module 720 includes an image semantic feature extraction module and an image saliency feature extraction module; the image semantic feature extraction module is used to input the target image into the semantic feature extraction module to obtain image semantic features; the image saliency feature extraction module is used to input the target image into the saliency feature extraction module to obtain image saliency features.

[0091] The feature fusion module 730 is used to perform weighted fusion on the image semantic features and image saliency features to obtain saliency weighted semantic features.

[0092] The output module 740 is used to input the saliency weighted semantic features into a fully connected layer and output the classification result of the target image.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0094] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0095] According to an embodiment of the present invention, the present invention further provides an electronic device and a readable storage medium.

[0096] Figure 8 A schematic block diagram of an electronic device 800 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0097] The electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0098] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0099] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as methods S101 to S104. For example, in some embodiments, methods S101 to S104 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the methods S101 to S104 described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute methods S101 - S104 in any other appropriate manner (eg, by means of firmware).

[0100] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0104] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0105] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0107] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image classification method based on a multi-scale attention mechanism, characterized in that: include: Acquire a target image, wherein the target image includes semantic information; Inputting the target image into a semantic feature extraction module to obtain image semantic features; and inputting the target image into a salient feature extraction module to obtain image salient features; The image semantic features are weightedly fused with the image saliency features to obtain saliency weighted semantic features; Inputting the saliency weighted semantic features into a fully connected layer, and outputting the classification result of the target image; The semantic feature extraction module includes four consecutive stages; wherein the first stage includes a Patch merging layer and a first DM-former module in sequence, wherein the first stage is used to receive a target image; the second stage includes a first Patch merging layer and a second DM-former module in sequence; the third stage includes a second Patch merging layer and a third DM-former module in sequence; the fourth stage includes a third Patch merging layer and a fourth DM-former module in sequence; the output of the fourth stage is used as an image semantic feature; The first DM-former module, the second DM-former module, the third DM-former module and the fourth DM-former module have the same structure, and sequentially include a multi-level residual structure and a residual block; The multi-level residual structure includes several improved residual structures, and the output results of each improved residual structure are added and fused, and the output result is used as the output result of the multi-level residual structure; each improved residual structure includes a deep convolution layer, a first LN layer and a multi-head self-attention module MHA in sequence; The deep convolution layer obtains input features, encodes the relative position information of the input features, adds and fuses the obtained encoding results with the input features, and outputs the relative position information encoding; the relative position information encoding is input into the first LN layer, the output result of the first LN layer is input into the multi-head self-attention module MHA, the output result of the multi-head self-attention module MHA is added and fused with the relative position information encoding to obtain the output result of the improved residual structure; The residual block includes a second LN layer and a multi-layer perceptron MLP in sequence; the second LN layer obtains the output result of the multi-level residual structure, and outputs the residual result through the multi-layer perceptron MLP, and adds and fuses the residual result with the output result of the multi-level residual structure to output as a semantic feature.

2. The method according to claim 1, characterized in that: In the multi-level residual structure, different improved residual structures have different scales of the depth convolutional layers.

3. The method according to claim 1, characterized in that The step of inputting the target image into a semantic feature extraction module to obtain image semantic features comprises: In the first stage, the target image is divided into blocks and the blocks are embedded using the Patch embedding layer to obtain the image block feature matrix. The image block feature matrix is ​​then input into the first DM-former module to output the first-level semantic features. In the second stage, the first-level semantic features are input into the first Patch merging layer, and the output of the first Patch merging layer is input into the second DM-former module to output the second-level semantic features; In the third stage, the second-level semantic features are input into the second Patch merging layer, and the output result of the second Patch merging layer is input into the third DM-former module to output the third-level semantic features; In the fourth stage, the three-level semantic features are input into the third Patch merging layer, and the output result of the third Patch merging layer is input into the fourth DM-former module to output the image semantic features.

4. The method according to claim 1, characterized in that The salient feature extraction module includes a saliency map extraction module and a ResNet50 model in sequence; The target image is input into the saliency map extraction module, and a saliency map is output; the saliency map is input into the ResNet50 model, and image saliency features are output.

5. The method according to claim 1, characterized in that The step of weighting and fusing the image semantic features with the image saliency features to obtain saliency weighted semantic features includes: Multiplying the image semantic feature with the image saliency feature to obtain a multiplied fusion feature; The multiplied fusion feature is added to the image semantic feature to obtain a saliency weighted semantic feature.

6. An image classification device based on a multi-scale attention mechanism, characterized in that: include: An acquisition module, used for acquiring a target image, wherein the target image includes semantic information; A feature extraction module, wherein the feature extraction module includes an image semantic feature extraction module and an image saliency feature extraction module; the image semantic feature extraction module is used to input the target image into the semantic feature extraction module to obtain image semantic features; the image saliency feature extraction module is used to input the target image into the saliency feature extraction module to obtain image saliency features; A feature fusion module, used for weighted fusion of the image semantic features and image saliency features to obtain saliency weighted semantic features; An output module, used for inputting the saliency weighted semantic features into a fully connected layer and outputting a classification result of a target image; The semantic feature extraction module includes four consecutive stages; wherein the first stage includes a Patch merging layer and a first DM-former module in sequence, wherein the first stage is used to receive a target image; the second stage includes a first Patch merging layer and a second DM-former module in sequence; the third stage includes a second Patch merging layer and a third DM-former module in sequence; the fourth stage includes a third Patch merging layer and a fourth DM-former module in sequence; the output of the fourth stage is used as an image semantic feature; The first DM-former module, the second DM-former module, the third DM-former module and the fourth DM-former module have the same structure, and sequentially include a multi-level residual structure and a residual block; The multi-level residual structure includes several improved residual structures, and the output results of each improved residual structure are added and fused, and the output result is used as the output result of the multi-level residual structure; each improved residual structure includes a deep convolution layer, a first LN layer and a multi-head self-attention module MHA in sequence; The deep convolution layer obtains input features, encodes the relative position information of the input features, adds and fuses the obtained encoding results with the input features, and outputs the relative position information encoding; the relative position information encoding is input into the first LN layer, the output result of the first LN layer is input into the multi-head self-attention module MHA, the output result of the multi-head self-attention module MHA is added and fused with the relative position information encoding to obtain the output result of the improved residual structure; The residual block includes a second LN layer and a multi-layer perceptron MLP in sequence; the second LN layer obtains the output result of the multi-level residual structure, and outputs the residual result through the multi-layer perceptron MLP, and adds and fuses the residual result with the output result of the multi-level residual structure to output as a semantic feature.

7. An electronic device comprising at least one processor; and a memory connected to the at least one processor in communication; characterized in that: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5.

Images