Image classification method, system, device and medium based on multi-dimensional attention

Abstract

The present invention provides a method for image classification based on multi-dimensional attention, which uses a picture data set to define the picture classification labels of each picture in the picture data set according to a preset scene; constructs a picture classification model, and adds a multi-dimensional attention mechanism to the picture classification model; uses the picture data set to train the picture classification model; samples and tests and verifies the picture classification model, and optimizes the picture classification model according to the test results; inputs the picture to be classified into the picture classification model to obtain the output classification result. The picture classification method redesigns and trains the deep learning model through the multi-dimensional attention mechanism module, and can achieve lightweight, fast, and high-accuracy mountain peak and valley scene image classification tasks, which can replace manual review and classification work, greatly save labor operation costs, improve image utilization efficiency, and enhance user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to an image classification method, system, device and storage medium based on multi-dimensional attention. Background Art

[0002] How to accurately understand the content of a wide variety of images and accurately classify them has become a research hotspot and is also a problem that needs to be solved urgently in practical application scenarios.

[0003] In recent years, the introduction of attention mechanism has injected new vitality into image classification tasks. However, these attention mechanism-based algorithms often only focus on the features of a certain dimension, have low accuracy, high labor costs for review, and low overall efficiency. Summary of the invention

[0004] In response to the problems in the prior art, the purpose of the present invention is to provide a method, system, device and medium for image classification based on multi-dimensional attention, which can redesign and train deep learning models through a multi-dimensional attention mechanism module, and can achieve lightweight, fast and high-accuracy image classification tasks.

[0005] The embodiment of the present invention provides a method for image classification based on multi-dimensional attention, comprising the following steps:

[0006] Constructing an image dataset, and defining image classification labels for each image in the image dataset according to a preset scenario;

[0007] Construct an image classification model, wherein a multi-dimensional attention mechanism is added to the image classification model;

[0008] Using the image dataset to train the image classification model;

[0009] Sampling test and verifying the image classification model, and optimizing the image classification model according to the test results;

[0010] The image to be classified is input into the image classification model to obtain the output classification result.

[0011] Optionally, the step of defining a picture classification label for each picture in the picture data set according to a preset scenario includes:

[0012] Defining a plurality of picture classification labels according to a preset scene, wherein the picture classification labels correspond to the preset scene;

[0013] Classifying the images in the image dataset according to the image classification labels;

[0014] The images that do not belong to the image classification label are defined as other labels.

[0015] Optionally, the steps of building an image classification model include:

[0016] The basic model structure of the image classification model is constructed using the wide resnet50 model pre-trained based on open source data;

[0017] A multi-dimensional attention mechanism module is added to the basic model structure, and the multi-dimensional attention mechanism module is used to adjust the intermediate feature map and the final feature map based on the multi-dimensional attention mechanism to obtain an image classification model.

[0018] Optionally, the wide resnet50 model includes a first convolution structure, a second convolution structure, a third convolution structure, a fourth convolution structure and a fifth convolution structure connected in series in sequence, and the multi-dimensional attention mechanism module is added after the third convolution structure and the fifth convolution structure respectively to obtain the image classification model.

[0019] Optionally, the step of training the image classification model includes:

[0020] Use weighted cross entropy to construct the model's loss function and calculate the model loss;

[0021] Using an adaptive moment estimation optimizer to update the weights of the image classification model, where the learning rate decays to 0.1 times the original rate every 20 epochs;

[0022] A first learning rate lr1 and a second learning rate lr2 are set, where lr1>lr2. The first learning rate lr2 is used to train the weights of the second convolution structure and the fourth convolution structure in the wide resnet50 model, and the second learning rate lr1 is used to train the weights of the multi-dimensional attention mechanism module and the last fully connected layer weight in the image classification model.

[0023] Optionally, the loss function of the image classification model is loss net It is expressed as:

[0024]

[0025] The output of the image classification model is Y = {y1, y2, ..., y N+1},W label is the weight of a specific label in the loss function, y label is the image category predicted by the network, N represents N categories, and the weight W = {w1, w2, ..., w N+1}, the label is the image classification label.

[0026] Optionally, the optimizing the image classification model includes:

[0027] Test and verify image classification results and analyze misclassified samples;

[0028] Supplementing corresponding positive and negative samples to the image dataset for the misclassified samples, and updating the image dataset;

[0029] The image classification model is iteratively trained based on the updated image dataset.

[0030] The embodiment of the present invention further provides a picture classification system based on multi-dimensional attention, which is applied to any of the above picture classification methods based on multi-dimensional attention, and the system includes:

[0031] A data set construction module is used to construct a picture data set and define a picture classification label for each picture in the picture data set according to a preset scenario;

[0032] A model building module, used to build an image classification model, in which a multi-dimensional attention mechanism is added;

[0033] A model training module, used to train the image classification model using the image data set, sample test and verify the image classification model, and optimize the image classification model according to the test results;

[0034] The image classification module is used to input the image to be classified into the image classification model to obtain the output classification result.

[0035] The embodiment of the present invention further provides a picture classification device based on multi-dimensional attention, characterized in that it includes:

[0036] processor;

[0037] a memory storing executable instructions of the processor;

[0038] Wherein, the processor is configured to perform the steps of the image classification method based on multi-dimensional attention by executing the executable instructions.

[0039] An embodiment of the present invention also provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the image classification method based on multi-dimensional attention.

[0040] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and cannot limit the scope of protection claimed by the technical solution of the present disclosure.

[0041] The multi-dimensional attention-based image classification method, system, device and medium of the present invention have the following beneficial effects:

[0042] The multi-dimensional attention-based image classification method of the present invention introduces a multi-dimensional attention mechanism module, redesigns and trains the deep learning model. This method is superior to existing image classification algorithms not only in accuracy, but also in speed and model size. It can achieve lightweight, fast, and high-accuracy mountain peak and valley scene image classification tasks, can replace manual review and classification work, greatly save labor operation costs, improve image utilization efficiency, and enhance user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0044] Figure 1 is a flowchart of a method for image classification based on multi-dimensional attention according to an embodiment of the present invention;

[0045] Figure 2 is a logical schematic diagram of a method for image classification based on multi-dimensional attention according to an embodiment of the present invention;

[0046] Figure 3 is a specific flow chart of a method for image classification based on multi-dimensional attention according to an embodiment of the present invention;

[0047] Figure 4 It is a module architecture diagram of a picture classification system based on multi-dimensional attention according to an embodiment of the present invention;

[0048] Figure 5 is a structural schematic diagram of a multi-dimensional attention-based image classification device according to an embodiment of the present invention;

[0049] Figure 6 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0051] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0052] like Figure 1 As shown, an embodiment of the present invention provides a method for image classification based on multi-dimensional attention, comprising the following steps:

[0053] S100: constructing a picture data set, and defining a picture classification label of each picture in the picture data set according to a preset scenario;

[0054] S200: constructing an image classification model, wherein a multi-dimensional attention mechanism is added to the image classification model;

[0055] S300: Using the image dataset to train the image classification model;

[0056] S400: Sampling test and verifying the image classification model, and optimizing the image classification model according to the test results;

[0057] S500: Inputting the image to be classified into the image classification model to obtain an output classification result.

[0058] In the method of this embodiment, the serial number of each step is only used to distinguish the steps, but not to limit the specific execution order of the steps. The execution order between the above steps can be adjusted and changed as needed.

[0059] like Figure 3 As shown, the image classification method based on multi-dimensional attention of the present invention specifically includes the following steps:

[0060] S100 builds a picture dataset and collects N pictures in various ways according to the label definition scene. total , and scale these images to a height and width size of H×W. Define N labels for the image classification labels of each image in the image dataset according to the preset scenario, and define the images that do not belong to the image classification labels as other labels. total The pictures are divided into three parts: training set N train , validation set N val And the test set N test .

[0061] S200 builds an image classification model, in which a multi-dimensional attention mechanism is added. The constructed image classification model structure is called Net resnet-mda The specific method is: Figure 2 As shown in S210, we first use the wide resnet50 model Net based on open source data pre-training resnet Construct the basic model structure of the image classification model, where the model has 50 network layers, and the resnet The model includes a first convolution structure conv1_x, a second convolution structure conv2_x, a third convolution structure conv3_x, a fourth convolution structure conv4_x and a fifth convolution structure conv5_x which are connected in series in sequence. The multi-dimensional attention mechanism module is added after the third convolution structure and the fifth convolution structure respectively to obtain the image classification model. The number of output nodes of the final fully connected layer is set to N+1, that is, the above-mentioned N labels and 1 other label. Figure 2 As shown, the Net resnet The model also includes average pooling and softmax layers (Average-pooling FC softmax) after each convolution structure to classify images based on the features extracted by the convolution structure, and obtain the probability Score_1, Score_2... of the image corresponding to each category Class.

[0062] As shown in S220, a multi-dimensional attention mechanism module (MDA module) is added to the basic model structure, and the multi-dimensional attention mechanism module is used to adjust the intermediate feature map and the final feature map based on the multi-dimensional attention mechanism to obtain an image classification model. The input feature map is feature input , the output feature map is:

[0063] feature output =feature input ×Attention widt h×Attention depth ,

[0064] Among them, Attention width and Attention depth Represent the wide and deep attention modules respectively.

[0065] Attention width =MDA width (feature input )

[0066] Attention depth =MDA depth(feature input )

[0067] like Figure 2 As shown, the above MDA module is introduced into Net resnet , add MDA modules after conv3_x and conv5_x to enhance the model feature extraction and representation capabilities. The new conv3_x and conv5_x are expressed as:

[0068] conv3_mda=MDA width (conv3-x)

[0069] conv5_mda=MDA width (conv5_x)

[0070] In another embodiment, S300 uses a training set N in the image dataset. train The steps of training the image classification model include:

[0071] As described in S310, a weighted cross entropy is used to construct a loss function of the model and calculate the model loss;

[0072] As described in S320, the weights of the image classification model are updated using an adaptive moment estimation optimizer, wherein the learning rate is decayed to 0.1 times the original value every 20 epochs;

[0073] As described in S330, train the model weights. Set a first learning rate lr1 and a second learning rate lr2, where lr1>lr2, use the smaller lr2 to train the existing weights of the model, and use the larger lr1 to train the weights of the multi-dimensional attention mechanism module and the weights of the fully connected layer of the picture classification, that is, use the first learning rate lr2 to train the weights of conv2_x and conv4_x in the wide resnet50 model, and use the second learning rate lr1 to train the weights of the multi-dimensional attention mechanism module and the weights of the last fully connected layer in the picture classification model.

[0074] Optionally, the loss function loss of the image classification model in S310 net It is expressed as:

[0075]

[0076] The output of the image classification model is Y = {y1, y2, ..., y N+1}, W is the weight of the image category in the loss function, and the weight W={w1, w2, …, w N+1},w label is the weight of a specific category label in the loss function, y labelis the image category predicted by the network, N represents N categories, and label is the image classification label.

[0077] In a preferred embodiment, the present invention performs sampling test and verification on the image classification model as described in S400, and optimizes the image classification model according to the test results. Specifically, a certain amount of images other than the above-mentioned image data set are randomly selected for testing, the image classification results are recorded, the image classification results are verified and checked, the misclassified samples are analyzed, and the accuracy, recall rate and other indicators are evaluated; the corresponding positive and negative samples are added to the image data set for the misclassified samples, some atypical samples are removed, the image data set is updated, and the cross entropy weight W is updated; the image classification model is iteratively trained based on the updated image data set. Repeat multiple rounds of data iteration model training until the accuracy of the model meets the requirements of online production.

[0078] Based on the TorchServe model server of Pytorch or the tf serving model server of tensorflow, the trained and optimized image classification model is packaged and deployed, and the service interface is developed in combination with Gunicorn and Flask frameworks. As described in S500, a large number of unlabeled images to be classified are input into the image classification model to obtain the output classification results.

[0079] like Figure 4 As shown, an embodiment of the present invention further provides a system for image classification based on multi-dimensional attention, comprising:

[0080] The data set construction module M100 is used to construct a picture data set and define a picture classification label of each picture in the picture data set according to a preset scenario;

[0081] A model building module M200 is used to build an image classification model, in which a multi-dimensional attention mechanism is added;

[0082] A model training module M300 is used to train the image classification model using the image data set, sample test and verify the image classification model, and optimize the image classification model according to the test results;

[0083] The picture classification module M400 is used to input the picture to be classified into the picture classification model to obtain the output classification result.

[0084] The present invention also provides a picture classification device based on multi-dimensional attention, characterized in that it includes:

[0085] processor;

[0086] a memory in which executable instructions for the processor are stored;

[0087] The processor is configured to execute the steps of image classification based on multi-dimensional attention in any one of the embodiments by executing executable instructions.

[0088] Refer to the following Figure 5 The electronic device 600 according to this embodiment of the present invention is described. Figure 5 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0089] like Figure 5 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0090] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the method section of the above-mentioned multi-dimensional attention-based image classification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0091] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202 , and may further include a read-only storage unit (ROM) 6203 .

[0092] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0093] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0094] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a client to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0095] An embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the method for image classification based on multi-dimensional attention are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is executed on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned method section of image classification based on multi-dimensional attention in this specification.

[0096] refer to Figure 6 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be executed on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0097] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0098] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0099] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the client computing device, partially on the client device, as a separate software package, partially on the client computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the client computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0100] In summary, by adopting the method, system, device and medium of multi-dimensional attention-based image classification of the present invention, by introducing a multi-dimensional attention mechanism module, redesigning and training the deep learning model, it is possible to achieve lightweight, fast, and high-accuracy mountain peak and valley scene image classification tasks, which can replace manual review and classification work, greatly save labor operation costs, improve image utilization efficiency, and enhance user experience.

[0101] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for image classification based on multi-dimensional attention, characterized in that: The following steps are involved: Constructing an image dataset, and defining image classification labels for each image in the image dataset according to a preset scenario; Construct an image classification model, wherein a multi-dimensional attention mechanism is added to the image classification model; Using the image dataset to train the image classification model; Sampling test and verifying the image classification model, and optimizing the image classification model according to the test results; Input the image to be classified into the image classification model to obtain an output classification result; The steps to build an image classification model include: The basic model structure of the image classification model is constructed using the wide resnet50 model pre-trained based on open source data; A multi-dimensional attention mechanism module is added to the basic model structure, and the multi-dimensional attention mechanism module is used to adjust the intermediate feature map and the final feature map based on the multi-dimensional attention mechanism to obtain an image classification model; The intermediate feature map is feature input , the final feature map is: feature output =feature input ×Attention width ×Attention depth , Among them, Attention width and Attention depth Represent the width and depth attention modules respectively; Attention width =MDA width (feature input ), MDA width Represents the width multi-dimensional attention mechanism module, Attention depth =MDA depth (feature input ), MDA depth Represents the deep multi-dimensional attention mechanism module.

2. The image classification method according to claim 1, characterized in that: The step of defining the image classification label of each image in the image data set according to the preset scenario includes: Defining a plurality of picture classification labels according to a preset scene, wherein the picture classification labels correspond to the preset scene; Classifying the images in the image dataset according to the image classification labels; The images that do not belong to the image classification label are defined as other labels.

3. The image classification method according to claim 1, characterized in that: The wide resnet50 model includes a first convolution structure, a second convolution structure, a third convolution structure, a fourth convolution structure and a fifth convolution structure which are connected in series in sequence. The multi-dimensional attention mechanism module is added after the third convolution structure and the fifth convolution structure respectively to obtain the image classification model.

4. The image classification method according to claim 3, characterized in that: The step of training the picture classification model includes: Use weighted cross entropy to construct the model's loss function and calculate the model loss; Using an adaptive moment estimation optimizer to update the weights of the image classification model, where the learning rate decays to 0.1 times the original rate every 20 epochs; Set the first learning rate lr1 and the second learning rate lr2, where lr1>lr2, use the second learning rate lr2 to train the weights of the second convolution structure and the fourth convolution structure in the wide resnet50 model, and use the first learning rate lr1 to train the weights of the multi-dimensional attention mechanism module and the last fully connected layer weight in the image classification model.

5. The image classification method according to claim 4, characterized in that: The loss function loss of the image classification model net It is expressed as: The output of the image classification model is Y = {y1, y2, ..., y N+1 }, the label is the image classification label, w label is the weight of a specific label in the loss function, y label is the image category predicted by the network, and N represents N categories.

6. The image classification method according to claim 1, characterized in that: Optimizing the image classification model includes: Test and verify image classification results and analyze misclassified samples; Supplementing corresponding positive and negative samples to the image dataset for the misclassified samples, and updating the image dataset; The image classification model is iteratively trained based on the updated image dataset.

7. A picture classification system, characterized in that: The image classification method according to any one of claims 1 to 6, wherein the system comprises: A data set construction module is used to construct a picture data set and define a picture classification label for each picture in the picture data set according to a preset scenario; A model building module, used to build an image classification model, in which a multi-dimensional attention mechanism is added; A model training module, used to train the image classification model using the image data set, sample test and verify the image classification model, and optimize the image classification model according to the test results; The image classification module is used to input the image to be classified into the image classification model to obtain the output classification result; The steps to build an image classification model include: The basic model structure of the image classification model is constructed using the wide resnet50 model pre-trained based on open source data; A multi-dimensional attention mechanism module is added to the basic model structure, and the multi-dimensional attention mechanism module is used to adjust the intermediate feature map and the final feature map based on the multi-dimensional attention mechanism to obtain an image classification model; The intermediate feature map is feature input , the final feature map is: feature output =feature input ×Attention width ×Attention depth , Among them, Attention width and Attention depth Represent the width and depth attention modules respectively; Attention width =MDA width (feature input ), MDA width Represents the width multi-dimensional attention mechanism module, Attention depth =MDA depth (feature input ), MDA depth Represents the deep multi-dimensional attention mechanism module.

8. A multi-dimensional attention-based image classification device, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to perform the steps of the multi-dimensional attention-based image classification method described in any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the image classification method based on multi-dimensional attention described in any one of claims 1 to 6 are implemented.

Images