Image classification method and device based on double-branch fusion network, equipment and medium
By using a dual-branch fusion network in image classification to extract and fuse multi-scale semantic features and spatial detail features, the problem of surface object information loss in the underlying surface classification of the convolutional neural network is solved and the classification accuracy is improved.
Patent Information
- Application Number
- CN202411528693.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-01-21
AI Technical Summary
When classifying the lower surface, the existing convolutional neural network gradually becomes smaller due to the feature map, resulting in loss of ground material information, reducing the classification accuracy.
The image classification method based on the dual-branch fusion network is used to extract multi-scale semantic features and spatial detail features through the semantic feature branch and the spatial information branch, and fuse them through the feature aggregation decoder to output the image classification results.
By fusing spatial detail features and semantic features, the loss of image information is reduced and classification accuracy is improved.
Smart Images
Figure CN119339154A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image classification, and in particular to an image classification method, device, equipment and medium based on a dual-branch fusion network. Background Art
[0002] In urban construction, the underlying surface is divided into different types according to different standards. However, because there are many types of objects and the boundaries of objects are not clear enough, it is difficult to automatically classify the underlying surface. At present, the underlying surface classification is mainly completed by convolutional neural networks. However, the classification accuracy of existing convolutional neural networks is low and it is difficult to meet the requirements. This is because the feature map of the convolutional neural network will gradually become smaller when classifying the underlying surface, resulting in the loss of object information and reducing the classification accuracy. Summary of the invention
[0003] The purpose of the embodiments of the present invention is to provide an image classification method, device, equipment and medium based on a dual-branch fusion network, which can fuse the spatial detail features and semantic features in the image, thereby reducing the loss of image information and improving classification accuracy.
[0004] To achieve the above object, an embodiment of the present invention provides an image classification method based on a dual-branch fusion network, comprising:
[0005] Get the image to be classified;
[0006] The image to be classified is input into a pre-built dual-branch fusion network to obtain an image classification result; wherein the dual-branch fusion network includes:
[0007] A semantic feature branch, used to extract semantic feature maps of N scales from the image to be classified, where N is an integer greater than 1;
[0008] A spatial information branch is used to extract spatial detail feature maps and edge detail information maps of N scales from the image to be classified;
[0009] A feature aggregation decoder is used to aggregate and enhance the semantic feature maps of the N scales, the spatial detail feature maps of the N scales and the edge detail information map, and output the image classification result.
[0010] As an improvement of the above solution, the semantic feature branch consists of a first convolutional layer, a pooling layer and N residual modules, then the semantic feature map is extracted in the following way:
[0011] Using the first convolution layer and the pooling layer to perform a convolution operation and downsampling on the image to be classified to obtain a first intermediate feature;
[0012] The first intermediate feature is sequentially downsampled through the N residual modules to obtain N semantic feature maps with different resolutions.
[0013] As an improvement of the above solution, the spatial information branch consists of a second convolutional layer and N+1 convolutional attention modules. Then, the spatial detail feature map is extracted in the following manner:
[0014] Using the second convolutional layer to perform a convolution operation on the image to be classified to obtain a second intermediate feature;
[0015] The second intermediate feature is sequentially passed through the N+1 convolutional attention modules to extract horizontal spatial attributes and vertical spatial attributes to obtain N spatial detail feature maps.
[0016] As an improvement of the above solution, the spatial information branch includes an edge guiding module for extracting the edge detail information map, specifically including:
[0017] Extracting spatial feature information from each of the spatial detail feature maps except the first layer and concatenating them to obtain edge information of multi-scale objects;
[0018] The edge information is processed by a convolution kernel and then upsampled to obtain an edge detail information map.
[0019] As an improvement of the above solution, after obtaining the edge detail information map, the method further includes:
[0020] Using a 1×1 convolution kernel, the feature channel dimension of the edge detail information map is adjusted to 1 to obtain an edge prediction map;
[0021] The edge prediction map is normalized by using a sigmoid function to obtain an edge weight for guiding the aggregation of the semantic feature map and the spatial detail feature map.
[0022] As an improvement of the above scheme, the feature aggregation decoder includes N state space semantic feature fusion modules and N directional convolution modules, wherein each state space semantic feature fusion module is used to generate a corresponding final fusion feature, specifically including:
[0023] Based on the edge weight, the semantic feature map and the corresponding spatial detail feature map are weighted and summed to obtain a preliminary fusion feature map; wherein the edge weight is calculated based on the edge detail information map;
[0024] The semantic feature map, the spatial detail feature map and the preliminary fusion feature map are summed to obtain a spatial attention weight;
[0025] The preliminary fusion feature map and the spatial attention weight are multiplied element by element and processed by a convolution kernel to obtain the final fusion feature.
[0026] As an improvement of the above solution, the N directional convolution modules are used to aggregate the N final fusion features layer by layer, wherein the layer by layer aggregation starts from the Nth layer and ends at the first layer, specifically including:
[0027] The Nth layer directional convolution module is used to upsample the final fusion features of the Nth layer to obtain the Nth preliminary aggregation features;
[0028] The N-1th layer directional convolution module is used to upsample the N-1th layer final fusion features and the N-th preliminary aggregation features to obtain the N-1th preliminary aggregation features;
[0029] Aggregate layer by layer until the first-layer directional convolution module is used to upsample the first-layer final fusion features and the second preliminary aggregation features to obtain the final aggregation features.
[0030] To achieve the above object, an embodiment of the present invention further provides an image classification device based on a dual-branch fusion network, comprising:
[0031] An image acquisition module, used to acquire images to be classified;
[0032] The image classification module is used to input the image to be classified into a pre-built dual-branch fusion network to obtain an image classification result; wherein the dual-branch fusion network includes:
[0033] A semantic feature branch, used to extract semantic feature maps of N scales from the image to be classified, where N is an integer greater than 1;
[0034] A spatial information branch is used to extract spatial detail feature maps and edge detail information maps of N scales from the image to be classified;
[0035] A feature aggregation decoder is used to aggregate and enhance the semantic feature maps of the N scales, the spatial detail feature maps of the N scales and the edge detail information map, and output the image classification result.
[0036] To achieve the above objectives, an embodiment of the present invention also provides an image classification device based on a dual-branch fusion network, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the image classification method based on the dual-branch fusion network as described in any of the above embodiments.
[0037] To achieve the above-mentioned purpose, an embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the image classification method based on the dual-branch fusion network as described in any of the above embodiments.
[0038] Compared with the prior art, the image classification method, device, equipment and medium based on the dual-branch fusion network provided in the embodiment of the present invention obtain an image to be classified; input the image to be classified into a pre-constructed dual-branch fusion network to obtain an image classification result; wherein the dual-branch fusion network includes a semantic feature branch, a spatial information branch and a feature aggregation decoder, the semantic feature branch is used to extract N-scale semantic feature maps from the image to be classified, wherein N is an integer greater than 1; the spatial information branch is used to extract N-scale spatial detail feature maps and edge detail information maps from the image to be classified; the feature aggregation decoder is used to aggregate and enhance the N-scale semantic feature maps, the N-scale spatial detail feature maps and the edge detail information maps, and output the image classification result. The embodiment of the present invention can extract spatial detail features and semantic features from the image to be classified and fuse them, thereby reducing the loss of image information and improving classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a flow chart of an image classification method based on a dual-branch fusion network provided by an embodiment of the present invention;
[0040] Figure 2 is a schematic diagram of the structure of a dual-branch fusion network provided by an embodiment of the present invention;
[0041] Figure 3 It is a structural schematic diagram of an image classification device based on a double-branch fusion network provided by an embodiment of the present invention;
[0042] Figure 4 It is a structural schematic diagram of an image classification device based on a dual-branch fusion network provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.
[0045] See also Figure 1 , is a flowchart of an image classification method based on a dual-branch fusion network provided by an embodiment of the present invention, including steps S1 to S2:
[0046] S1, obtain the image to be classified;
[0047] S2. Input the image to be classified into a pre-built dual-branch fusion network to obtain a classification result; wherein the dual-branch fusion network includes:
[0048] A semantic feature branch, used to extract semantic feature maps of N scales from the image to be classified, where N is an integer greater than 1;
[0049] A spatial information branch is used to extract spatial detail feature maps and edge detail information maps of N scales from the image to be classified;
[0050] A feature aggregation decoder is used to aggregate and enhance the semantic feature maps of the N scales, the spatial detail feature maps of the N scales and the edge detail information map, and output the image classification result.
[0051] It is worth noting that the image classification method based on the dual-branch fusion network described in the embodiment of the present invention can be used for underlying surface classification, and can also be used for classification of other images, which is not limited here.
[0052] Further, taking the classification of the lower surface as an example, the image to be classified in step S1 may be a drone image of the target area.
[0053] Exemplarily, before step S1, the method further includes step S01 and step S02:
[0054] S01. Construct a data set, wherein the data set includes a training set and a validation set.
[0055] Exemplarily, the collected drone images are fully classified and labeled according to the underlying surface of the target area, a label file corresponding to each drone image is obtained, and a data set is constructed. Further, the data set is divided into a training set and a validation set according to a certain ratio, wherein the training set can be used to train the dual-branch fusion network, and the validation set can be used to verify whether the performance of the trained dual-branch fusion network meets the requirements.
[0056] S02. Perform data preprocessing on the data set, including data enhancement and image cropping.
[0057] It is understandable that data augmentation can increase the complexity of the data set and expand the number of data sets, thereby providing sufficient data support for the training of the dual-branch fusion network. Furthermore, image cropping includes cropping paired drone images and their corresponding label files according to a certain window size and step size. Image cropping can remove redundant parts in the image and ensure the consistency of the size of each image in the data set, which is convenient for model design and training.
[0058] Further, in step S2, the dual-branch fusion network is an image classification model constructed by an embodiment of the present invention, including a semantic feature branch, a spatial information branch and a feature aggregation decoder, wherein the semantic feature branch is used to generate multi-scale semantic features, the spatial information branch is used to extract the underlying multi-scale spatial detail features and edge detail information of the image to be classified, and generate edge predictions, and the feature aggregation decoder is used to aggregate, enhance and interpret the features generated by the above dual branches (semantic feature branch and spatial information branch) to obtain high-precision semantic predictions.
[0059] Exemplarily, the dual-branch fusion network can test and optimize the model through back-propagation and optimization algorithms, and use semantic loss and edge loss to guide the optimization of network parameters respectively.
[0060] Compared with the prior art, the embodiments of the present invention can reduce the differences between features and better restore the category information of ground objects by extracting the spatial detail features and semantic features of ground objects separately and fusing them, thereby achieving accurate classification of ground objects with limited data information.
[0061] As an optional implementation, the semantic feature branch consists of a first convolutional layer, a pooling layer and N residual modules, then the semantic feature map is extracted in the following manner:
[0062] Using the first convolution layer and the pooling layer to perform a convolution operation and downsampling on the image to be classified to obtain a first intermediate feature;
[0063] The first intermediate feature is sequentially downsampled through the N residual modules to obtain N semantic feature maps with different resolutions.
[0064] For example, when extracting semantic information, in order to balance the effectiveness and efficiency of feature extraction and to reduce computational costs, the residual network-18 can be used as the backbone architecture of the semantic feature branch. Figure 2 , is a schematic diagram of the structure of a dual-branch fusion network provided by an embodiment of the present invention. In this embodiment, the value of N is 4. Figure 2It can be seen that the semantic feature branch includes the first convolutional layer, the pooling layer and four residual modules (ResBlock), and each residual module performs two downsampling operations. Finally, the semantic feature branch generates multi-resolution semantic feature maps, which are represented as F1, F2, F3 and F4 respectively.
[0065] It can be understood that by generating multi-resolution semantic feature maps, semantic information can be extracted from different scales and resolutions of the image to be classified. Usually, different levels of the neural network will produce feature maps of different resolutions. The feature maps of the bottom layer have higher resolution but capture local information. The feature maps of the high layer have lower resolution but represent global semantic information.
[0066] As an optional implementation, the spatial information branch is composed of a second convolutional layer and N+1 convolutional attention modules, then the spatial detail feature map is extracted by:
[0067] Using the second convolutional layer to perform a convolution operation on the image to be classified to obtain a second intermediate feature;
[0068] The second intermediate feature is sequentially passed through the N+1 convolutional attention modules to extract horizontal spatial attributes and vertical spatial attributes to obtain N spatial detail feature maps.
[0069] See also Figure 2 , the spatial information branch includes the second convolutional layer, the edge-guidance feature module (EFD) and five convolutional attention modules (CBAM).
[0070] For example, in the spatial information branch, a directional convolution module can be used to capture the spatial attributes of the object in the horizontal and vertical directions to obtain accurate spatial detail information, and a shallow spatial information branch is constructed by stacking directional convolution modules to extract the low-level multi-phase and multi-scale spatial detail feature maps of the target object, which are represented as S1, S2, S3 and S4 (such as Figure 2 shown).
[0071] As an optional implementation manner, the spatial information branch includes an edge guiding module, which is used to extract the edge detail information map, specifically including:
[0072] Extracting spatial feature information from each of the spatial detail feature maps except the first layer and concatenating them to obtain edge information of multi-scale objects;
[0073] The edge information is processed by a convolution kernel and then upsampled to obtain an edge detail information map.
[0074] It is worth noting that, considering that the first-layer spatial detail feature map S1 has a large amount of insignificant edge information, using these edges to promote subsequent feature fusion will inevitably lead to more errors. The embodiment of the present invention selects spatial detail feature maps other than the first layer to extract edge information, that is, selects spatial detail feature maps S2, S3 and S4, upsamples and splices them to eliminate the influence of heterogeneity between categories and obtain edge information of multi-scale targets.
[0075] Specifically, a 1x1 convolution operation is first performed on the spatial detail feature map S2, and then batch normalization is applied, and an activation function is used to output the spatial feature information; a 1x1 convolution operation is then performed on the spatial detail feature map S3, and then batch normalization is applied, and an activation function is used to output the spatial feature information; finally, a 1x1 convolution operation is performed on the spatial detail feature map S4, and then batch normalization is applied, and an activation function is used to output the spatial feature information. After that, the spatial feature information obtained above is spliced to obtain the edge information of the multi-scale target, which is upsampled after being processed by a 3×3 convolution kernel to obtain the edge detail information map F. e , which is expressed by formula as shown in formula (1):
[0076] F e =Up4×(Conv 3×3 (concat((Up2(S2), Up4(S3), Up8(S4))))) (1)
[0077] Among them, F e Represents edge detail information map; Conv 3×3 Represents 3×3 convolution kernel processing; concat represents a concatenation operation; Up4 represents a four-fold bilinear upsampling operation; Up2 represents a two-fold bilinear upsampling operation; Up8 represents an eight-fold bilinear upsampling operation; S2 represents the second-layer spatial detail feature map; S3 represents the third-layer spatial detail feature map; S4 represents the fourth-layer spatial detail feature map.
[0078] As an optional implementation manner, after obtaining the edge detail information map, the method further includes:
[0079] Using a 1×1 convolution kernel, the feature channel dimension of the edge detail information map is adjusted to 1 to obtain an edge prediction map;
[0080] The edge prediction map is normalized by using a sigmoid function to obtain an edge weight for guiding the aggregation of the semantic feature map and the spatial detail feature map.
[0081] For example, first, the edge detail information map F eAdaptive average pooling is performed to keep its resolution consistent with the other two feature maps; then a 1×1 convolution kernel is used to adjust the number of feature channels to 1 to obtain the edge prediction map F. pe ; Then use the sigmoid function (σ) for normalization to get the edge weight F W , which is expressed by formula as shown in formula (2):
[0082] F W =σ(Conv 1×1 (Avgpool(F e )))=σ(F pe ) (2)
[0083] Among them, F W represents edge weight; σ represents sigmoid function; Conv 1×1 Indicates 1×1 convolution kernel processing; Avgpool indicates adaptive average pooling; F e Indicates edge detail information map; F pe Represents the edge prediction map.
[0084] As an optional implementation, the feature aggregation decoder includes N state space semantic feature fusion modules and N directional convolution modules, wherein each state space semantic feature fusion module is used to generate a corresponding final fusion feature, specifically including:
[0085] Based on the edge weight, the semantic feature map and the corresponding spatial detail feature map are weighted and summed to obtain a preliminary fusion feature map; wherein the edge weight is calculated based on the edge detail information map;
[0086] The semantic feature map, the spatial detail feature map and the preliminary fusion feature map are summed to obtain a spatial attention weight;
[0087] The preliminary fusion feature map and the spatial attention weight are multiplied element by element and processed by a convolution kernel to obtain the final fusion feature.
[0088] See also Figure 2 The feature aggregation decoder includes four state space semantic feature fusion modules (SSSF, StateSpatial-Semantic Feature Module) and four direction convolution modules (OCM, OrientationConvolution Module). In the feature aggregation decoder, a semantic feature map F is established. i , spatial detail feature map S i and edge detail information graph F e The relationship between them is used to generate the final fusion feature FS i.
[0089] Specifically, using the edge weight F W For the semantic feature map F i and spatial detail feature map S i The fused features are weighted respectively to obtain semantic features and spatial features guided by boundary information; further, in order to ensure the diversity of features and the trainability of the network, the embodiment of the present invention also introduces a residual structure, that is, the fused features are summed through a 3×3 convolution kernel to obtain a preliminary fused feature map It is expressed in formula as shown in formula (3):
[0090]
[0091] in, Represents the preliminary fusion feature map of the i-th layer; Conv 3×3 represents 3×3 convolution kernel processing; F i represents the semantic feature map of the i-th layer; F W represents the edge weight; S i Represents the spatial detail feature map of the i-th layer; Represents element-by-element addition operation; Represents an element-wise multiplication operation.
[0092] Furthermore, since the fusion initially has a lot of redundant information, in order to focus the network on the spatial position of the object of interest, the embodiment of the present invention further generates spatial attention weights. i , spatial detail feature map S i And the initial fusion feature map The summation process is performed, that is, element-by-element addition is performed to obtain the spatial attention weight, thereby realizing the complementarity between features at different levels, that is, realizing the complementarity between the spatial information branch and the semantic feature branch. It is worth noting that the spatial information branch is responsible for extracting low-level multi-phase and multi-scale spatial detail features and edge information features, which include the spatial attributes and edge details of the object; the semantic feature branch generates multi-scale semantic feature maps, which capture the high-level semantic information of the object; the embodiment of the present invention improves the overall performance of the dual-branch fusion network by realizing the complementarity between low-level spatial detail features and high-level semantic features.
[0093] Furthermore, the preliminary fusion feature map and the spatial attention weight are multiplied element by element and processed by a 3×3 convolution kernel to obtain the final fusion feature FS i , which is expressed by formula as shown in formula (4)
[0094]
[0095] Among them, FS i Represents the final fusion feature of the i-th layer; Conv 3×3 Represents 3×3 convolution kernel processing; SA (SpatialAttention) represents the spatial attention module; represents the initial fusion feature map of the i-th layer; F i represents the semantic feature map of the i-th layer; S i Represents the spatial detail feature map of the i-th layer; Represents element-wise multiplication operation; Represents element-wise addition operation.
[0096] As an optional implementation manner, the N directional convolution modules are used to aggregate the N final fusion features layer by layer, wherein the layer by layer aggregation starts from the Nth layer and ends at the first layer, specifically including:
[0097] The Nth layer directional convolution module is used to upsample the final fusion features of the Nth layer to obtain the Nth preliminary aggregation features;
[0098] The N-1th layer directional convolution module is used to upsample the N-1th layer final fusion features and the N-th preliminary aggregation features to obtain the N-1th preliminary aggregation features;
[0099] Aggregate layer by layer until the first-layer directional convolution module is used to upsample the first-layer final fusion features and the second preliminary aggregation features to obtain the final aggregation features.
[0100] Exemplarily, the directional convolution module is composed of directional convolution modules, which are used to aggregate the final fusion features of different scales layer by layer to obtain the final aggregated features. Figure 2 , the bottom layer (layer N, such as Figure 2 The input of the directional convolution module of the fourth layer (the fourth layer in the image) is the final fusion feature of the current layer, and the input of the directional convolution modules of the remaining layers is the output of the directional convolution module of the previous layer (preliminary aggregated features) and the final fusion feature of the current layer. Specifically, the input of the directional convolution module of the fourth layer is the fourth layer final fusion feature FS4, and the output is the fourth preliminary aggregated feature AF4; the input of the directional convolution module of the third layer is the third layer final fusion feature FS3 and the fourth preliminary aggregated feature AF4, and the output is the third preliminary aggregated feature AF3; the input of the directional convolution module of the second layer is the second layer final fusion feature FS2 and the third preliminary aggregated feature AF3, and the output is the second preliminary aggregated feature AF2; the input of the directional convolution module of the first layer is the first layer final fusion feature FS1 and the second preliminary aggregated feature AF2, and the output is the final aggregated feature AF1.
[0101] Furthermore, during the training process of the dual-branch fusion network, the accuracy evaluation index can be used to measure the training effect of the model. Specifically, the dual-branch fusion network to be trained is used to classify the objects in the image to be classified, and the underlying surface in the classification result is extracted; then, a confusion matrix is constructed based on the regional real classification vector graphics data and the prediction results of the dual-branch network to perform accuracy analysis on the classification results of the underlying surface at different scales and evaluate the performance of the dual-branch fusion network.
[0102] Exemplarily, the accuracy evaluation index can adopt the Kappa coefficient, OA (Overall Accuracy) or MIoU (Mean Intersection over Union), where the Kappa coefficient is an indicator for measuring classification accuracy, which is obtained by multiplying the total number of pixels in all true surface classifications by the sum of the diagonals of the confusion matrix, subtracting the product of the total number of true surface pixels of a certain class and the total number of classified pixels in the class, and then dividing it by the square of the total number of pixels minus the product of the total number of true surface pixels of a certain class and the total number of classified pixels in the class, and summing the result of all classes.
[0103] Compared with the prior art, the image classification method based on the dual-branch fusion network provided in the embodiment of the present invention obtains an image to be classified; inputs the image to be classified into a pre-constructed dual-branch fusion network to obtain an image classification result; wherein the dual-branch fusion network includes a semantic feature branch, a spatial information branch and a feature aggregation decoder, the semantic feature branch is used to extract N-scale semantic feature maps from the image to be classified, wherein N is an integer greater than 1; the spatial information branch is used to extract N-scale spatial detail feature maps and edge detail information maps from the image to be classified; the feature aggregation decoder is used to aggregate and enhance the N-scale semantic feature maps, the N-scale spatial detail feature maps and the edge detail information maps, and output the image classification result. The embodiment of the present invention can extract spatial detail features and semantic features from the image to be classified and fuse them, thereby reducing the loss of image information and improving classification accuracy.
[0104] See also Figure 3 The embodiment of the present invention further provides an image classification device 10 based on a dual-branch fusion network, comprising:
[0105] An image acquisition module 11 is used to acquire an image to be classified;
[0106] The image classification module 12 is used to input the image to be classified into a pre-built dual-branch fusion network to obtain an image classification result; wherein the dual-branch fusion network includes:
[0107] A semantic feature branch, used to extract semantic feature maps of N scales from the image to be classified, where N is an integer greater than 1;
[0108] A spatial information branch is used to extract spatial detail feature maps and edge detail information maps of N scales from the image to be classified;
[0109] A feature aggregation decoder is used to aggregate and enhance the semantic feature maps of the N scales, the spatial detail feature maps of the N scales and the edge detail information map, and output the image classification result.
[0110] The image classification device based on the dual-branch fusion network provided in the embodiment of the present invention can implement all the process steps of the image classification method based on the dual-branch fusion network described in the above embodiment. The functions of each module and unit in the device and the technical effects achieved are respectively the same as the functions and technical effects achieved by the image classification method based on the dual-branch fusion network described in the above embodiment. The specific implementation method will not be repeated here.
[0111] See also Figure 4 The embodiment of the present invention further provides an image classification device 20 based on a dual-branch fusion network, comprising a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21. When the processor 21 executes the computer program, the steps in the above-mentioned image classification method embodiment based on the dual-branch fusion network are implemented, for example Figure 1 or, the processor 21 implements the functions of each module in the above-mentioned device embodiments when executing the computer program.
[0112] The image classification device based on the dual-branch fusion network can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The image classification device based on the dual-branch fusion network may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the schematic diagram is only an example of an image classification device based on a dual-branch fusion network, and does not constitute a limitation on the image classification device based on the dual-branch fusion network. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the image classification device based on the dual-branch fusion network may also include input and output devices, network access devices, buses, etc.
[0113] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the image classification device based on the dual-branch fusion network, and uses various interfaces and lines to connect the various parts of the entire image classification device based on the dual-branch fusion network.
[0114] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the image classification device based on the dual-branch fusion network by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc.; the data storage area can store data created according to the use of the controller, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0115] Wherein, if the module integrated in the image classification device based on the dual-branch fusion network is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.
[0116] Compared with the prior art, the image classification device, equipment and medium based on the dual-branch fusion network provided in the embodiment of the present invention obtains an image to be classified; inputs the image to be classified into a pre-constructed dual-branch fusion network to obtain an image classification result; wherein the dual-branch fusion network includes a semantic feature branch, a spatial information branch and a feature aggregation decoder, the semantic feature branch is used to extract N-scale semantic feature maps from the image to be classified, wherein N is an integer greater than 1; the spatial information branch is used to extract N-scale spatial detail feature maps and edge detail information maps from the image to be classified; the feature aggregation decoder is used to aggregate and enhance the N-scale semantic feature maps, the N-scale spatial detail feature maps and the edge detail information maps, and output the image classification result. The embodiment of the present invention can extract spatial detail features and semantic features from the image to be classified and fuse them, thereby reducing the loss of image information and improving classification accuracy.
[0117] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. An image classification method based on a dual-branch fusion network, characterized in that: include: Get the image to be classified; The image to be classified is input into a pre-built dual-branch fusion network to obtain an image classification result; wherein the dual-branch fusion network includes: A semantic feature branch, used to extract semantic feature maps of N scales from the image to be classified, where N is an integer greater than 1; A spatial information branch is used to extract spatial detail feature maps and edge detail information maps of N scales from the image to be classified; A feature aggregation decoder is used to aggregate and enhance the semantic feature maps of the N scales, the spatial detail feature maps of the N scales and the edge detail information map, and output the image classification result.
2. The image classification method based on a dual-branch fusion network as claimed in claim 1, characterized in that: The semantic feature branch consists of a first convolutional layer, a pooling layer and N residual modules. Then, the semantic feature map is extracted in the following way: Using the first convolution layer and the pooling layer to perform a convolution operation and downsampling on the image to be classified to obtain a first intermediate feature; The first intermediate feature is sequentially downsampled through the N residual modules to obtain N semantic feature maps with different resolutions.
3. The image classification method based on a dual-branch fusion network as claimed in claim 1, characterized in that: The spatial information branch consists of the second convolutional layer and N+1 convolutional attention modules. Then, the spatial detail feature map is extracted in the following way: Using the second convolutional layer to perform a convolution operation on the image to be classified to obtain a second intermediate feature; The second intermediate feature is sequentially passed through the N+1 convolutional attention modules to extract horizontal spatial attributes and vertical spatial attributes to obtain N spatial detail feature maps.
4. The image classification method based on a dual-branch fusion network as claimed in claim 1, characterized in that: The spatial information branch includes an edge guidance module, which is used to extract the edge detail information map, specifically including: Extracting spatial feature information from each of the spatial detail feature maps except the first layer and concatenating them to obtain edge information of multi-scale objects; The edge information is processed by a convolution kernel and then upsampled to obtain an edge detail information map.
5. The image classification method based on a dual-branch fusion network as claimed in claim 4, characterized in that: After obtaining the edge detail information map, the method further includes: Using a 1×1 convolution kernel, the feature channel dimension of the edge detail information map is adjusted to 1 to obtain an edge prediction map; The edge prediction map is normalized by using a sigmoid function to obtain an edge weight for guiding the aggregation of the semantic feature map and the spatial detail feature map.
6. The image classification method based on a dual-branch fusion network as claimed in claim 1, characterized in that: The feature aggregation decoder includes N state space semantic feature fusion modules and N directional convolution modules, wherein each state space semantic feature fusion module is used to generate a corresponding final fusion feature, specifically including: Based on the edge weight, the semantic feature map and the corresponding spatial detail feature map are weighted and summed to obtain a preliminary fusion feature map; wherein the edge weight is calculated based on the edge detail information map; The semantic feature map, the spatial detail feature map and the preliminary fusion feature map are summed to obtain a spatial attention weight; The preliminary fusion feature map and the spatial attention weight are multiplied element by element and processed by a convolution kernel to obtain the final fusion feature.
7. The image classification method based on a dual-branch fusion network as claimed in claim 6, characterized in that: The N directional convolution modules are used to aggregate the N final fusion features layer by layer, wherein the layer by layer aggregation starts from the Nth layer and ends at the first layer, specifically including: The Nth layer directional convolution module is used to upsample the final fusion features of the Nth layer to obtain the Nth preliminary aggregation features; The N-1th layer directional convolution module is used to upsample the N-1th layer final fusion features and the N-th preliminary aggregation features to obtain the N-1th preliminary aggregation features; Aggregate layer by layer until the first-layer directional convolution module is used to upsample the first-layer final fusion features and the second preliminary aggregation features to obtain the final aggregation features.
8. An image classification device based on a dual-branch fusion network, characterized in that: include: An image acquisition module, used to acquire images to be classified; The image classification module is used to input the image to be classified into a pre-built dual-branch fusion network to obtain an image classification result; wherein the dual-branch fusion network includes: A semantic feature branch, used to extract semantic feature maps of N scales from the image to be classified, where N is an integer greater than 1; A spatial information branch is used to extract spatial detail feature maps and edge detail information maps of N scales from the image to be classified; A feature aggregation decoder is used to aggregate and enhance the semantic feature maps of the N scales, the spatial detail feature maps of the N scales and the edge detail information map, and output the image classification result.
9. An image classification device based on a dual-branch fusion network, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the image classification method based on the dual-branch fusion network as claimed in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the image classification method based on the dual-branch fusion network according to any one of claims 1 to 7.