A fine-grained image classification method and system, terminal device and storage medium

Abstract

The present invention discloses a fine-grained image classification method and system, terminal equipment and storage medium. The fine-grained image classification method comprises the following steps: obtaining a picture to be classified; inputting the picture to be classified into a pre-constructed diversity feature complementary fusion network, and using the diversity feature complementary fusion network to classify the picture; and obtaining the classification result of the picture. The method uses a diversity feature complementary fusion network to classify pictures, automatically and effectively captures subtle differences in specific areas by capturing the significant parts of the pictures, and classifies the pictures based on the subtle differences, effectively improving the classification efficiency and classification accuracy when classifying pictures. The method does not require a bounding box and can be trained end-to-end.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification methods, and relates to a fine-grained image classification method and system, a terminal device and a storage medium. Background Art

[0002] The purpose of fine-grained classification is to make more detailed sub-classifications of images belonging to the same basic category. For example, distinguishing wild birds from cars. Due to the subtle inter-class differences and large intra-class differences between categories, it is difficult to capture the subtle differences in specific areas for classification. Although deep learning has promoted the research of computer vision tasks, its application in fine-grained classification is still less satisfactory, which is caused by the difficulty in finding information-rich areas and extracting discriminative features from them. For categories with diverse postures such as birds, the classification situation is even worse.

[0003] Therefore, how to let CNN locate distinguishable parts and learn distinguishable features is an important problem that needs to be solved. The key to fine-grained classification is to develop automatic methods to accurately identify informative areas in images. Some previous methods have utilized fine-grained human annotations, such as annotations of bird parts in bird classification. Although this can achieve good results, it consumes a lot of manpower and requires relatively professional knowledge, which makes these methods less practical.

[0004] Therefore, many people use unsupervised learning schemes to locate informative areas. They eliminate the need for manual annotations, but lack a mechanism to ensure that the model focuses on the correct area, which usually leads to decreased accuracy. Unsupervised learning mainly moves in two directions. One is based on partial features. These methods usually consist of two different sub-networks. Specifically, the localization sub-network with an attention mechanism is designed to locate the discriminative part and follows the classification sub-network for identification. Due to the specially designed attention module and loss function, the network is difficult to optimize. The other is based on high-order information. These methods believe that first-order information is not sufficient to model the differences in images, but use high-order information to encode the distinction. The limitations of these methods are that they take up a lot of computing resources and have poor interpretability. Summary of the invention

[0005] In response to the problems existing in the prior art, the present invention provides a fine-grained image classification method and system, a terminal device and a storage medium, so as to achieve the network's attention to potential features, automatically and effectively capture subtle differences in specific areas, and classify images based on subtle differences, effectively improving the classification efficiency and classification accuracy when classifying images.

[0006] The present invention is achieved through the following technical solutions:

[0007] A fine-grained image classification method comprises the following steps:

[0008] S1: Get the image to be classified;

[0009] S2: inputting the image to be classified into a pre-constructed diversity feature complementary fusion network, and using the diversity feature complementary fusion network to classify the image;

[0010] S3: Obtain classification results of the image.

[0011] Preferably, the diversity feature complementary fusion network includes significant and latent feature modules and a feature exchange fusion module.

[0012] Preferably, classifying the image using the diversity feature complementary fusion network includes the following steps:

[0013] S201, passing the image to be classified into the diversity feature complementary fusion network, after entering the image in the network to extract features, passing through the salient and latent feature modules, outputting salient features, and obtaining latent features based on the salient features, and the latent features continue to be passed to the next layer of the network;

[0014] S202, passing the significant features extracted in step S201 into the convolution module for dimensional transformation;

[0015] S203, passing the features of the transformed dimensions in step S202 into a feature exchange and fusion module, performing feature fusion, and enhancing the diversity of the features to obtain the final diversity features;

[0016] S204, performing a global average pooling operation on the final diversity features obtained in step S203, reducing the dimension of the features of the transformed dimension through a linear layer, and optimizing the diversity feature complementary fusion network using a cross entropy loss function, performing distance measurement, and completing the classification of the image.

[0017] Preferably, the step S201 is specifically as follows:

[0018] In step S201, the image to be classified is sequentially passed into three saliency and potential feature modules, and the output is X s1 , X s2 and X s3 Three significant features, and suppress the significant features to obtain the potential feature X p , and X p Passed to the next layer of the network, specifically:

[0019] If the feature size of the input module of the diversity feature complementary fusion network is X∈RC×W×H , where C, W, and H represent the channel, width, and height of the feature respectively; in the module, X is equally divided into m equal sub-features along the width direction, and the width of the sub-feature is T, where T = W / m, and the sub-feature is represented by X i ∈R C×T×H , i∈[1,m]; use 1×1 convolution Conv() and ReLU() operations to process the sub-features to obtain feature S i , the feature S i It is expressed as:

[0020] S i =ReLU(Conv(X i ))∈R 1×T×H (20)

[0021] For the obtained feature S i Perform a generalized average pooling operation, specifically:

[0022]

[0023] In the formula, H D is the height of the feature, W D is the width of the feature, is the cumulative sum of all values ​​in the current dimension, and p is the parameter of generalized average pooling;

[0024] The saliency parameter b is obtained by the generalized average pooling i ';

[0025] b′ i =GEM(S i )∈R 1×1 (twenty two)

[0026] The obtained significance parameter B'=(b'1,...b' m )∈R 1×m×1 Perform normalization operation to obtain the significance factor, which is:

[0027]

[0028] In the formula, B=(b1,…b m )∈R 1×m is a significant factor;

[0029] The normalized significance factor B = (b1,…b m )∈R 1×m The weighted sum is performed with the original input feature X to obtain the significant part of the feature; before obtaining the significant part of the feature, the significant factor B = (b1, ...b m )∈R1×m×1 Perform size transformation, specifically B = repeat(B)∈R 1×W×1 , we get X i The significant feature X in s :

[0030] X s =(X⊙(α*B)) (24)

[0031] Among them, α is a hyperparameter, which is used to adjust the significance parameter; ⊙ represents the matrix dot product;

[0032] The feature S i Perform generalized average pooling again to obtain feature t′ i :

[0033] t′ i =GEM(S i )∈R 1×1 (25)

[0034] Use the softmax function to transform the feature t′ i After normalization, we get the feature T = (t1…t m )∈R 1×m×1 ; For the feature T = (t1…t m )∈R 1×m×1 Suppress the significance of and obtain the remaining potential features:

[0035]

[0036] β∈[0,1] is a hyperparameter; by removing the significant parameters in the latent features and then performing the resizing process, specifically T=repeat(T)∈R 1×W×1 ; Get the potential feature X p :

[0037] X p =X⊙T (27)

[0038] ⊙ represents matrix dot product.

[0039] Preferably, the step S202 is specifically as follows:

[0040] The feature X obtained in step S201 s1 ,X s2 ,X s2 Input into the 1×1 convolution layer Conv1×1(), perform dimension change processing, and obtain the features X o1 ,X o2 ,X o3 :

[0041] Xo1 =Conv1×1(X s1 ); (28)

[0042] X o2 =Conv1×1(X s2 ); (29)

[0043] X o3 =Conv1×1(X s3 ). (30)

[0044] Preferably, the step S203 is specifically as follows:

[0045] If the dimension-transformed feature obtained in step S202 is Where C, W, H represent the channel, width and height of the feature respectively; In the input feature exchange fusion module, the input features are mapped and the feature size is changed. o1 , X o2 Get the mapping features X1∈R C×l and X2∈R C×l , l = W × H; respectively calculate the weight matrix ω of the mapping feature o1 and ω o2 , where ω o1 ∈R C ×C ,ω o2 ∈R C×C , the weight matrix of the feature is calculated as follows:

[0046] First, in X o1 and Perform bilinear calculations between them to obtain a bilinear matrix Then the bilinear matrix Negative calculation is performed and processed using the softmax function to obtain the weight function. The calculation formula is as follows:

[0047]

[0048] ω ij is the weight matrix obtained by the function, c is the number of channels;

[0049] The difference information of two features is extracted by performing a subtraction operation on the weight matrix, specifically:

[0050] ω s =|ω o1 -ω o2 | (32)

[0051] || indicates absolute value;

[0052] The obtained weight matrix ω o1 and ω o2 With the input feature X o1 , X o2 Multiply to get the diversity feature:

[0053] X′1=ω s X1∈R C×l , X'2=ω s X2∈R C×l (33)

[0054] Mapping the diversity features back to the dimensions of the input features

[0055] If the input feature set after dimension transformation is I = {X o1 ,X o2 ,X o2 ,…X on}, the diversity characteristics of the feature set after the dimension transformation are:

[0056]

[0057] The diversity feature Y of the feature set after the transformation dimension is adopted oi The input features Enhance to get the final input features The final diversity feature is expressed as:

[0058] Y i =X oi +λ*Y oi (35)

[0059] Y i is the final diversity feature, X oi is the input feature and λ is a hyperparameter.

[0060] Preferably, the step S204 is specifically as follows:

[0061] The final diversity feature Y is calculated using the cross entropy loss function i The classification loss is first i Perform global average pooling and full connection dimension transformation operations, specifically: f i =FC(GAP(Y i )),i=1,2,3;

[0062] After global average pooling and full connection dimension transformation operations, the Softmax function is used to calculate the classification probability, specifically:

[0063]

[0064] In the formula, f i c is the eigenvector f i The cth term of , C is the number of categories, where T i =[T i 1 ,T i 2 ,…,T i c ], T i c Indicates that the i-th feature in the vector belongs to class c;

[0065] Then use the cross entropy loss function to calculate the loss:

[0066] L i =-y T log(T i ) (37)

[0067] Among them, y is the true label of the input image, T i is the predicted classification score vector, and the final optimization loss function is:

[0068]

[0069] The predicted average of all features is taken as the final prediction result.

[0070] A picture classification system, comprising:

[0071] Data acquisition module, used to collect pictures;

[0072] The recognition and classification module is used to input the image to be classified into a preset diversity feature complementary fusion network, and classify the image using the diversity feature complementary fusion network;

[0073] The result output module is used to obtain the classification result of the image.

[0074] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor runs the above-mentioned fine-grained image classification method when executing the computer program.

[0075] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the fine-grained image classification method described above is executed.

[0076] Compared with the prior art, the present invention has the following beneficial technical effects:

[0077] A fine-grained image classification method uses a diversity feature complementary fusion network to classify images. The diversity feature complementary fusion network includes a Significance and Potential Feature Module (SPFM), which enhances the salient features of the feature map, while mining the remaining potential features and ensuring the correlation between features. The Feature Exchange Fusion Module (FEFM) is used to fuse the extracted inter-channel difference information with features of different scales and depths to supplement the semantic information in features of different scales, achieve information complementarity, improve feature diversity, and enhance network performance. By capturing the salient parts of the image, the subtle differences in specific areas are automatically and effectively captured, and the image is classified based on the subtle differences, effectively improving the classification efficiency and classification accuracy when classifying images. This method does not require a bounding box and can be trained end-to-end. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0079] Figure 1 A schematic diagram of a process of a fine-grained image classification method in an embodiment of the present invention;

[0080] Figure 2 It is a schematic diagram of the framework of the diversity feature complementary fusion network in the present invention;

[0081] Figure 3 Schematic diagram of the structure of the significant potential feature module in the present invention;

[0082] Figure 4 The step of classifying pictures by the diversity feature complementary fusion network in the present invention;

[0083] Figure 5 It is a structural schematic diagram of a picture classification system in the present invention. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0085] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0087] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0088] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0089] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0090] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0091] Example 1

[0092] like Figure 1 As shown, a fine-grained image classification method includes the following steps:

[0093] S1: Get the image to be classified;

[0094] S2: inputting the image to be classified into a pre-constructed diversity feature complementary fusion network, and using the diversity feature complementary fusion network to classify the image;

[0095] like Figure 2 As shown in FIG. 1 , the diversity feature complementary fusion network includes three Significance and Potential Feature Modules (SPFMs), and the three SPFMs are located at the ends of residual module three, residual module four, and residual module five, respectively.

[0096] like Figure 3 As shown in the figure, SPFM includes a feature block structure that can be divided along the width of the feature; a 1x1 convolution layer that mainly performs a dimensionality reduction process on the feature; and a generalized-mean pooling layer (Generalized-mean Pooling, GeM) containing a first parameter and a second parameter. The first parameter is used to obtain significant features and the second parameter is used to obtain latent features. The first parameter is 3 and the second parameter is 7.

[0097] The diversity feature complementary fusion network also includes a feature exchange fusion module (Feature Exchange Fusion Module, FEFM). The feature exchange fusion module is located after the saliency and potential feature modules. It mainly processes the saliency features extracted from the saliency and potential feature modules. After passing through this module, the diversity features are obtained, which plays a role in enhancing the richness of features.

[0098] FEFM includes a feature difference module (FDM) and a feature fusion module. The extracted significant features are passed into the proposed feature exchange and fusion module. The features first pass through the feature difference module. In this module, the difference information between features can be obtained. After obtaining the feature difference information, the information is supplemented to the corresponding feature parts through feature fusion to achieve the effect of feature diversity and enhance the robustness of the network.

[0099] FDM consists of 1x1 convolution and matrix subtraction. 1x1 convolution is used to transform the dimension of the incoming salient features and adjust their dimensions to be consistent. The salient features after dimension adjustment are used to generate a weight matrix of size CXC according to the dimensions, where C represents the feature channel. The generated weight matrix is ​​then used for matrix subtraction to obtain differential features, and finally feature fusion is used to obtain diversity features.

[0100] SPFM includes two steps: (1) the features obtained in this stage are significantly enhanced to obtain significant features; (2) the significant features after feature enhancement are suppressed to obtain latent features. The significant features are extracted and passed to the feature exchange and fusion module, and then the latent features after significant suppression are passed to the network to further mine the category information in the latent features.

[0101] S3: Obtain classification results of the image.

[0102] like Figure 4 As shown, the diversity feature complementary fusion network classifies the image including the following steps:

[0103] S201, passing the image to be classified into the diversity feature complementary fusion network, after entering the image in the network to extract features, passing through the salient and latent feature modules, outputting salient features, and obtaining latent features based on the salient features, and the latent features continue to be passed to the next layer of the network;

[0104] Step S201 is specifically as follows:

[0105] In step S201, the image to be classified is sequentially passed into three saliency and potential feature modules, and the output is X s1 , X s2 and X s3 Three significant features, and suppress the significant features to obtain the potential feature X p , and X p Passed to the next layer of the network, specifically:

[0106] If the feature size of the input module of the diversity feature complementary fusion network is X∈R C×W×H , where C, W, and H represent the channel, width, and height of the feature respectively; in the module, X is equally divided into m equal sub-features along the width direction, and the width of the sub-feature is T, where T = W / m, and the sub-feature is represented by X i ∈R C×T×H , i∈[1,m]; use 1×1 convolution Conv() and ReLU() operations to process the sub-features to obtain feature S i , the feature S i It is expressed as:

[0107] S i =ReLU(Conv(X i ))∈R 1×T×H (39)

[0108] For the obtained feature S i Perform a generalized average pooling operation, specifically:

[0109]

[0110] In the formula, H D is the height of the feature, W D is the width of the feature, is the cumulative sum of all values ​​in the current dimension, and p is the parameter of generalized average pooling;

[0111] The saliency parameter b′ is obtained by the generalized average pooling i ;

[0112] b′ i =GEM(S i )∈R 1×1 (41)

[0113] The obtained significance parameter B'=(b'1,...b' m )∈R 1×m×1 Perform normalization operation to obtain the significance factor, which is:

[0114]

[0115] In the formula, B=(b1,…b m )∈R 1×m is a significant factor;

[0116] The normalized significance factor B = (b1,…b m )∈R 1×m The weighted sum is performed with the original input feature X to obtain the significant part of the feature; before obtaining the significant part of the feature, the significant factor B = (b1, ...b m )∈R 1×m×1 Perform size transformation, specifically B = repeat(B)∈R 1×W×1 , we get X i The significant feature X in s :

[0117] X s =(X⊙(α*B)) (43)

[0118] Among them, α is a hyperparameter, which is used to adjust the significance parameter; ⊙ represents the matrix dot product;

[0119] The feature S i Perform generalized average pooling again to obtain feature t′ i :

[0120] t′ i =GEM(S i )∈R 1×1 (44)

[0121] Use the softmax function to transform the feature t′ i After normalization, we get the feature T = (t1…t m )∈R 1×m×1 ; For the feature T = (t1…t m )∈R 1×m×1 Suppress the significance of and obtain the remaining potential features:

[0122]

[0123] β∈[0,1] is a hyperparameter; by removing the significant parameters in the latent features and then performing the resizing process, specifically T=repeat(T)∈R 1×W×1 ; Get the potential feature X p :

[0124] X p =X⊙T (46)

[0125] ⊙ represents matrix dot product.

[0126] S202, passing the significant features extracted in step S201 into the convolution module for dimensional transformation;

[0127] Step S202 is specifically as follows:

[0128] The feature X obtained in step S201 s1 ,X s2 ,X s2 Input into the 1×1 convolution layer Conv1×1(), perform dimension change processing, and obtain the features X o1 ,X o2 ,X o3 :

[0129] X o1 =Conv1×1(X s1 ) (47)

[0130] X o2 =Conv1×1(X s2 ) (48)

[0131] X o3 =Conv1×1(Xs3 ) (49)

[0132] S203, passing the features of the transformed dimensions in step S202 into a feature exchange and fusion module, performing feature fusion, and enhancing the diversity of the features to obtain the final diversity features;

[0133] Step S203 is specifically as follows:

[0134] If the dimension-transformed feature obtained in step S202 is Where C, W, H represent the channel, width and height of the feature respectively; In the input feature exchange fusion module, the input features are mapped and the feature size is changed. o1 , X o2 Get the mapping features X1∈R C×l and X2∈R C×l , l = W × H; respectively calculate the weight matrix ω of the mapping feature o1 and ω o2 , where ω o1 ∈R C ×C ,ω o2 ∈R C×C , the weight matrix of the feature is calculated as follows:

[0135] First, in X o1 and Perform bilinear calculations between them to obtain a bilinear matrix Then the bilinear matrix Negative calculation is performed and processed using the softmax function to obtain the weight function. The calculation formula is as follows:

[0136]

[0137] ω ij is the weight matrix obtained by the function, c is the number of channels;

[0138] The difference information of two features is extracted by performing a subtraction operation on the weight matrix, specifically:

[0139] ω s =|ω o1 -ω o2 | (51)

[0140] || indicates absolute value;

[0141] The obtained weight matrix ω o1 and ω o2 With the input feature X o1, X o2 Multiply to get the diversity feature:

[0142] X′1=ω s X1∈R C×l , X'2=ω s X2∈R C×l (52)

[0143] Mapping the diversity features back to the dimensions of the input features

[0144] If the input feature set after dimension transformation is I = {X o1 ,X o2 ,X o2 ,…X on}, the diversity characteristics of the feature set after the dimension transformation are:

[0145]

[0146] The diversity feature Y of the feature set after the transformation dimension is adopted oi The input features Enhance to get the final input features The final diversity feature is expressed as:

[0147] Y i =X oi +λ*Y oi (54)

[0148] Y i is the final diversity feature, X oi is the input feature and λ is a hyperparameter.

[0149] S204, performing a global average pooling operation on the final diversity features obtained in step S203, reducing the dimension of the features of the transformed dimension through a linear layer, and optimizing the diversity feature complementary fusion network using a cross entropy loss function, performing distance measurement, and completing the classification of the image.

[0150] Step S204 is specifically as follows:

[0151] The final diversity feature Y is calculated using the cross entropy loss function i The classification loss is first i Perform global average pooling and full connection dimension transformation operations, specifically: f i =FC(GAP(Y i )),i=1,2,3;

[0152] After global average pooling and full connection dimension transformation operations, the Softmax function is used to calculate the classification probability, specifically:

[0153]

[0154] In the formula, f i c is the eigenvector f i The cth term of , C is the number of categories, where T i =[T i 1 ,T i 2 ,…,T i c ], T i c Indicates that the i-th feature in the vector belongs to class c;

[0155] Then use the cross entropy loss function to calculate the loss:

[0156] L i =-y T log(T i ) (56)

[0157] Among them, y is the true label of the input image, T i is the predicted classification score vector, and the final optimization loss function is:

[0158]

[0159] The predicted average of all features is taken as the final prediction result.

[0160] like Figure 5 As shown, a picture classification system includes:

[0161] The data acquisition module 100 is used to acquire pictures;

[0162] The recognition and classification module 200 is used to input the image to be classified into a preset diversity feature complementary fusion network, and classify the image using the diversity feature complementary fusion network;

[0163] The result output module 300 is used to obtain the classification result of the image.

[0164] An embodiment of the present invention further provides a terminal device, the terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned image classification process is implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0165] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0166] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0167] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0168] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0169] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements each step or part of the process in the picture classification process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement each step in the above-mentioned infrared video action recognition process. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0170] Example 2

[0171] A fine-grained classification method based on a diverse feature complementary fusion network is implemented in the following steps:

[0172] Step 1: First, the classified image is passed into the diversity feature complementary fusion network (DFCF). After the image in the network is extracted, it passes through the significance and potential feature module (SPFM) to output the significance feature X. s , and the significant suppression of the significant features is used as the latent feature X p Continue to pass to the next layer of the network;

[0173] Step 2: The significant feature X extracted in step 1 s They are respectively passed into the 1x1 convolution Conv() module for a one-dimensional transformation;

[0174] Step 3: The features of the transformed dimensions in step 2 are passed into the feature exchange fusion module (FEFM) to perform feature fusion and enhance feature diversity.

[0175] Step 4: Perform global average pooling on the output diversity features, and transform the input feature dimensions through the final linear layer. The input feature dimensions are too large and the amount of calculation is huge. The linear layer is used to reduce the dimension and refine the features for subsequent processing. The cross entropy loss function is used to optimize the network for distance measurement to obtain the classification result.

[0176] Specific implementation methods of step 1:

[0177] Given an image, we pay more attention to its salient objects and describe its actions and attributes when observing the image. In the diversity feature complementary fusion network (DFCF), the main function of the Significance and potential feature module (SPFM) is to enhance the salient features of the feature map, while mining the remaining potential features and ensuring the correlation between features.

[0178] Using ResNet-50 as the basic framework, the saliency and potential feature module (SPFM) is flexibly inserted into the stage 3, 4, and 5 layers of the network. The incoming image is output through (SPFM) respectively X s1 , X s2 and X s3 There are three significant features in total, and the significant features are suppressed to obtain the latent feature X p It is passed to the next layer of the network, such as the potential features obtained from stage 3 are suppressed and passed to stage 4.

[0179] Assume that the characteristic size of the input module is X∈R C×W×H , where C, W, and H represent the channel, width, and height of the feature respectively. In the module, X is divided into m equal sub-features along the width direction, and the width of the sub-feature is T, where T = W / m. Each sub-feature can be represented as X i ∈R C×T×H , i∈[1,m], and then use a 1×1 convolution Conv() and ReLU() operation to process each sub-feature to obtain feature S i :

[0180] S i=ReLU(Conv(X i ))∈R 1×T×H (58)

[0181] Then the obtained feature S i Perform a generalized average pooling (GEM) operation to obtain the significance parameter b′ representing the significant feature i , so we can use b′ i To represent X i Compared with global average pooling, generalized average pooling (GEM) has an additional adjustable parameter p. When p is 1, generalized average pooling is global average pooling. When p tends to infinity, it is global maximum pooling:

[0182]

[0183] Therefore, the saliency parameter b′ is obtained by generalized average pooling (GEM) i for:

[0184] b′ i =GEM(S i )∈R 1×1 (60)

[0185] Here, the parameter p of generalized average pooling (GEM) is set to p=7. When extracting significant features, p is set to a slightly larger value of 7 because GEM considers the parameters of the entire dimension, so the significant features obtained are more representative. When p is larger, it will be closer to the global maximum pooling, which will affect the model performance. Then use the softmax function to obtain the significant parameters B'=(b'1,…b' m )∈R 1×m×1 Perform normalization operation

[0186]

[0187] The normalized significance factor B = (b1,…b m )∈R 1×m By weighted summing with the original input feature X, the significant part of the feature can be obtained. Before this, due to the mismatch in size, it is necessary to first m )∈R 1×m×1 Perform size conversion, B = repeat(B)∈R 1×W×1 , from which we can get X i The significant feature X in s :

[0188] X s=(X⊙(α*B)) (62)

[0189] α is a hyperparameter for adjusting the significance parameter;

[0190] ⊙ represents matrix dot product.

[0191] The feature S extracted above i Generalized average pooling (GEM) is performed again, except that the parameter p of (GEM) is set to p=3, thus the feature t′ can be obtained i :

[0192] t′ i =GEM(S i )∈R 1×1 (63)

[0193] The method of obtaining significant features is the same as that of using the softmax function to perform feature t′ i After normalization, we get the feature T = (t1…t m )∈R 1×m×1 In order to obtain the latent features, it is necessary to suppress the features T = (t1…t m )∈R 1×m×1 The saliency of these features is used so that the network can focus on the remaining potential features and continuously explore the potential information in the features:

[0194]

[0195] β∈[0,1] is a hyperparameter that represents the degree of significant suppression. The smaller the value, the stronger the suppression. By removing the significant parameters in the latent factors and then performing the resizing process T=repeat(T)∈R 1×W×1 , from which we can get the potential feature X p :

[0196] X p =X⊙T (65)

[0197] ⊙ represents matrix dot product. The proposed SPFM module can be expressed as: SPFM(X) = (X o ,X p ).

[0198] Step 2 Specific implementation method:

[0199] The obtained feature X s1 ,X s2 ,X s2 Input into the 1×1 convolution layer Conv1×1(), perform a dimension change process, and obtain the features X o1 ,X o2 ,Xo3 :

[0200] X o1 =Conv1×1(X s1 ) (66)

[0201] X o2 =Conv1×1(X s2 ) (67)

[0202] X o3 =Conv1×1(X s3 ) (68)

[0203] Step 3 Specific implementation method:

[0204] The input features are represented as Where C, W, and H represent the channel, width, and height of the feature respectively. First, the input feature is mapped to change the size of the feature. o1 , X o2 Get the mapping features X1∈R C×l and X2∈R C×l , l = W × H. Calculate the weight matrix ω o1 and ω o2 (ω o1 ∈R C×C ,ω o2 ∈R C×C ), the feature matrix is ​​calculated as follows. First, in X o1 and Perform bilinear calculations between them to obtain a bilinear matrix Then it is negative and processed using the softmax function to obtain the weight function. The calculation formula is as follows:

[0205]

[0206] In order to extract the difference between two features, such difference information is extracted by subtracting the weight matrix:

[0207] ω s =|ω o1 -ω o2 | (70)

[0208] || represents the absolute value, and the obtained weight matrix is ​​the difference information of the features. Through subtraction, the correlation between each other is suppressed and the difference between different features is enhanced. The obtained weight matrix ω o1 and ω o2 With the initial input feature X o1 , X o2 Multiply to get the diversity feature:

[0209] X′1=ω s X1∈R C×l , X'2=ω s X2∈R C×l (71)

[0210] Map the obtained diversity features back to the size of the input features for subsequent processing Assume that the input feature set is I = {X o1 ,X o2 ,X o2 ,…X on}, X oi The diversity characteristics can be expressed as:

[0211]

[0212] This gives the final feature representation:

[0213] Y i =X oi +λ*Y oi (73)

[0214] λ is a hyperparameter, and changing its value can increase the degree of feature diversity.

[0215] Step 4 Specific implementation method:

[0216] During the training phase, the cross entropy loss function is used to calculate each final diversity feature Y i The classification loss is firstly performed on the features by global average pooling (GAP) and full connection (FC) dimension transformation operations, specifically f i =FC(GAP(Y i )), i = 1, 2, 3. After processing, the Softmax function is used to calculate the classification probability:

[0217]

[0218] where f i c is the eigenvector f i The cth term of , C is the number of categories, where T i =[T i 1 ,T i 2 ,…,T i c ], T i c Indicates that the i-th feature in the vector belongs to class c. Then use the cross entropy loss function to calculate the loss:

[0219] L i =-y T log(T i ) (75)

[0220] Where y is the true label of the input image, represented by a one-hot vector, T i is the predicted classification score vector, and the final optimization loss function is:

[0221]

[0222] Where N = 3 is the number of significant features. During testing, the predicted average of all features is taken as the final prediction result. By optimizing the above loss function, the proposed network model can be trained end-to-end, and the final classification result can be obtained after one training.

[0223] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fine-grained image classification method, characterized in that: The following steps are involved: S1: Get the image to be classified; S2: inputting the image to be classified into a pre-constructed diversity feature complementary fusion network, and using the diversity feature complementary fusion network to classify the image; S3: Obtain the classification result of the image; The diversity feature complementary fusion network includes saliency and potential feature modules and a feature exchange fusion module; Using the diversity feature complementary fusion network to classify the image includes the following steps: S201, passing the image to be classified into the diversity feature complementary fusion network, after entering the image in the network to extract features, passing through the salient and latent feature modules, outputting salient features, and obtaining latent features based on the salient features, and the latent features continue to be passed to the next layer of the network; S202, passing the significant features extracted in step S201 into the convolution module for dimensional transformation; S203, passing the features of the transformed dimensions in step S202 into a feature exchange and fusion module, performing feature fusion, and enhancing the diversity of the features to obtain the final diversity features; S204, performing a global average pooling operation on the final diversity features obtained in step S203, reducing the dimension of the features of the transformed dimension through a linear layer, optimizing the diversity feature complementary fusion network using a cross entropy loss function, performing distance measurement, and completing the classification of the image; The step S201 is specifically as follows: In step S201, the image to be classified is sequentially passed into three saliency and potential feature modules, and the output is X s1 , X s2 and X s3 Three significant features, and suppress the significant features to obtain the potential feature X p , and X p Passed to the next layer of the network, specifically: If the feature size of the input module of the diversity feature complementary fusion network is X∈R C×W×H , where C, W, and H represent the channel, width, and height of the feature respectively; in the module, X is equally divided into m equal sub-features along the width direction, and the width of the sub-feature is T, where T = W / m, and the sub-feature is represented by X i ∈R C×T×H , i∈[1,m]; use 1×1 convolution Conv() and ReLU() operations to process the sub-features to obtain feature S i , the feature S i It is expressed as: S i =ReLU(Conv(X i ))∈R 1×T×H (1) For the obtained feature S i Perform a generalized average pooling operation, specifically: In the formula, H D is the height of the feature, W D is the width of the feature, is the cumulative sum of all values ​​in the current dimension, and p is the parameter of generalized average pooling; The saliency parameter b is obtained by the generalized average pooling i '; b i '=GEM(S i ) ∈R 1×1 (3) The obtained significance parameter B'=(b'1,…b' m )∈R 1×m×1 Perform normalization operation to obtain the significance factor, which is: In the formula, B=(b1,…b m )∈R 1×m is a significant factor; The normalized significance factor B = (b1,…b m )∈R 1×m The weighted sum is performed with the original input feature X to obtain the significant part of the feature; before obtaining the significant part of the feature, the significant factor B = (b1, ...b m )∈R 1 ×m×1 Perform size transformation, specifically B = repeat(B)∈R 1×W×1 , we get X i The significant feature X in s : X s =(X⊙(α*B)) (5) Among them, α is a hyperparameter, which is used to adjust the significance parameter; ⊙ represents the matrix dot product; The feature S i Perform generalized average pooling again to obtain feature t' i : t' i =GEM(S i )∈R 1×1 (6) Use the softmax function to perform feature t' i After normalization, we get the feature T = (t1…t m )∈R 1×m×1 ; For the feature T = (t1…t m )∈R 1×m×1 Suppress the significance of and obtain the remaining potential features: β∈[0,1] is a hyperparameter; by removing the significant parameters in the latent features and then performing the resizing process, specifically T=repeat(T)∈R 1×W×1 ; Get the potential feature X p : X p =X⊙T (8) ⊙ represents matrix dot product.

2. A fine-grained image classification method according to claim 1, characterized in that: The step S202 is specifically as follows: The feature X obtained in step S201 s1 ,X s2 ,X s2 Input into the 1×1 convolution layer Conv1×1(), perform dimension change processing, and obtain the features X o1 ,X o2 ,X o3 : X o1 =Conv1×1(X s1 ); (9) X o2 =Conv1×1(X s2 ); (10) X o3 =Conv1×1(X s3 ) (11)。 3. A fine-grained image classification method according to claim 2, characterized in that: The step S203 is specifically as follows: If the dimension-transformed feature obtained in step S202 is Where C, W, H represent the channel, width and height of the feature respectively; In the input feature exchange fusion module, the input features are mapped and the feature size is changed. o1 , X o2 Get the mapping features X1∈R C×l and X2∈R C×l , l = W × H; respectively calculate the weight matrix ω of the mapping feature o1 and ω o2 , where ω o1 ∈R C×C ,ω o2 ∈R C×C , the weight matrix of the feature is calculated as follows: First, in X o1 and Perform bilinear calculations between them to obtain a bilinear matrix Then the bilinear matrix Negative calculation is performed and the softmax function is used to process the weight function. The calculation formula is as follows: ω ij is the weight matrix obtained by the function, c is the number of channels; The difference information of two features is extracted by performing a subtraction operation on the weight matrix, specifically: oh s =|ω o1 -oh o2 | (13) | | indicates absolute value; The obtained weight matrix ω o1 and ω o2 With the input feature X o1 , X o2 Multiply them together to get the diversity feature: X'1=ω s X1∈R C×l ,X'2=ω s X2∈R C×l (14) Map the diversity features back to the dimensions of the input features If the input feature set after dimension transformation is I = {X o1 ,X o2 ,X o2 ,…X on }, the diversity characteristics of the feature set after the dimension transformation are: The diversity feature Y of the feature set after the transformation dimension is adopted oi The input features Enhance to get the final input features The final diversity feature is expressed as: AND i =X oi +λ*Y oi (16) Y i is the final diversity feature, X oi is the input feature and λ is a hyperparameter.

4. The fine-grained image classification method according to claim 1, characterized in that: The step S204 is specifically as follows: The final diversity feature Y is calculated using the cross entropy loss function i The classification loss is first i Perform global average pooling and full connection dimension transformation operations, specifically: f i =FC(GAP(Y i )),i=1,2,3; After global average pooling and full connection dimension transformation operations, the Softmax function is used to calculate the classification probability, specifically: In the formula, f i c is the eigenvector f i The cth term of , C is the number of categories, where T i =[T i 1 ,T i 2 ,…,T i c ], T i c Indicates that the i-th feature in the vector belongs to class c; Then use the cross entropy loss function to calculate the loss: L i =-y T log(T i ) (18) Among them, y is the true label of the input image, T i is the predicted classification score vector, and the final optimization loss function is: The predicted average of all features is taken as the final prediction result.

5. A picture classification system, characterized in that: Used to implement a fine-grained image classification method as described in claim 1, and comprising: Data acquisition module, used to collect pictures; The recognition and classification module is used to input the image to be classified into a preset diversity feature complementary fusion network, and classify the image using the diversity feature complementary fusion network; The result output module is used to obtain the classification result of the image.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the fine-grained image classification method according to any one of claims 1 to 4 is executed.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the fine-grained image classification method according to any one of claims 1 to 4 is executed.

Images