A Chinese multi-label recognition method based on multi-scale fusion and attention mechanism

Abstract

The application discloses a Chinese multi-label recognition method based on multi-scale fusion and an attention mechanism, and relates to the technical field of image recognition. The Chinese multi-label recognition method comprises the following steps: constructing a Chinese recognition network model, wherein the Chinese recognition network model comprises a convolutional neural network, an attention module and a classification module; the convolutional neural network comprises a first convolution block, a second convolution block, a third convolution block, a fourth convolution block and a fifth convolution block which are sequentially connected; the attention module comprises a global attention module and two local attention modules; obtaining a Chinese picture; extracting information of the Chinese picture through the convolutional neural network; extracting attention features of feature maps output by the third convolution block and the fourth convolution module through the two local attention modules respectively; extracting attention features of a feature map output by the fifth convolution block through the global attention module; splicing extraction results of the two local attention modules and the extraction result of the global attention module to obtain a spliced vector; and classifying the spliced vector through the classification module to obtain a food category in the Chinese picture.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Chinese food recognition, and particularly relates to a Chinese food multi-label recognition method based on multi-scale fusion and attention mechanism. BACKGROUND

[0002] Accurate food recognition is the basis for mining other information of food (such as calorie prediction, food material prediction and nutrition component prediction). If a food recognition model is mature, it can be used for doctor's supervision and improvement of patient's diet, diet supervision of healthy people or automatic recognition of food websites.

[0003] When a picture contains multiple dishes, each dish only appears in a local area of the picture, and the food materials in the dish are rich, the food materials block each other, and are also affected by light and complex background. How to quickly and accurately extract dish information from the picture has become a challenge of the current picture recognition task. SUMMARY

[0004] The application aims to provide a Chinese food multi-label recognition method based on multi-scale fusion and attention mechanism, which can recognize all foods contained in a picture at one time; the network model adopted has small memory occupation, fast training speed, and can improve the accuracy of Chinese food picture recognition.

[0005] The technical scheme provided by the application is as follows:

[0006] A Chinese food multi-label recognition method based on multi-scale fusion and attention mechanism, comprising:

[0007] A Chinese food recognition network model is constructed, which comprises a convolutional neural network, an attention module and a classification module;

[0008] The convolutional neural network comprises a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block and a fifth convolutional block connected in sequence; the attention module comprises a global attention module and two local attention modules, and the two local attention modules correspond to the third convolutional block and the fourth convolutional block one by one;

[0009] A Chinese food picture is obtained by using a camera device, and information extraction is performed on the Chinese food picture by using the convolutional neural network;

[0010] The two local attention modules are used to extract attention features of feature maps output by the third convolutional block and the fourth convolutional module, respectively; and

[0011] The global attention module is used to extract attention features of a feature map output by the fifth convolutional block;

[0012] The extraction results of the two local attention modules and the extraction result of the global attention module are spliced to obtain a spliced vector;

[0013] The spliced vector is classified by the classification module to obtain a food category in the Chinese food picture.

[0014] Preferably, the attention feature of the feature map output by the third convolutional block is extracted by the local attention module, including the following steps:

[0015] The feature map subjected to the channel attention mechanism is evenly cut into four parts;

[0016] Attention calculation is respectively performed on the four parts to obtain an attention score of each part;

[0017] After the attention score of each part is pooled, the pooling results of each part are spliced to obtain the attention feature of the feature map output by the third convolutional block.

[0018] Preferably, the pooling results of each part of the feature map are obtained by the following relationship:

[0019] pooling(F i )=λ*Maxpool(F i )+Avgpool(F i );

[0020] In the formula, F i represents the feature map after the calculation attention score of the i-th part, Maxpool(F i ) represents the result of the maximum pooling of F i , Avgpool(F i ) represents the result of the average pooling of F i , pooling(F i ) represents the result of the pooling operation of F i , and λ represents a weight parameter.

[0021] Preferably, the method of extracting the attention feature of the feature map output by the fourth convolutional block is the same as the method of extracting the attention feature of the feature map output by the third convolutional block.

[0022] Preferably, the size of the picture input by the first convolutional block is 224×224×3.

[0023] Preferably, the third convolutional block includes four convolutional layers, and the output channel of each convolutional layer is 128 and the convolution kernel size is 3×3.

[0024] Preferably, the fourth convolutional block comprises four convolutional layers, each having 256 output channels and a 3*3 convolution kernel size.

[0025] Preferably, the fifth convolutional block comprises four convolutional layers, each having 512 output channels and a 3*3 convolution kernel size.

[0026] Preferably, the convolutional neural network adopts ResNet18.

[0027] Preferably, the classification module classifies the spliced vector, comprising the following steps:

[0028] using a linear layer to perform category prediction on the spliced vector:

[0029] out = OW T +b;

[0030] wherein out is a category prediction value with a size of C*1, C represents the number of categories; W is a weight parameter to be learned during training, and b is a bias;

[0031] The maximum value in the out vector is recorded as max (out), and the out vector is traversed, if there is an element value greater than 0.85*max (out), the Chinese food picture contains the food category corresponding to the element value, which is represented as:

[0032] out[result] > 0.85*max(out);

[0033] wherein result is an output food category set, and out[result] represents the probability value of the output category.

[0034] The beneficial effects of the present application are:

[0035] The Chinese multi-label recognition method based on multi-scale fusion and attention mechanism provided by the present application can recognize multiple foods in a picture at one time, has a small network model parameter quantity, occupies a small memory, has a fast training speed, and can reach an mAP value of 0.956 on a food set data set. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 It is a framework diagram of the Chinese food recognition network model.

[0037] Figure 2 It is a channel attention mechanism schematic diagram in the CBAM.

[0038] Figure 3 It is a spatial attention mechanism schematic diagram in the CBAM.

[0039] Figure 4 A local attention mechanism diagram for the present application.

[0040] Figure 5 A model training loss diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described in conjunction with the accompanying drawings, so that those skilled in the art can implement it according to the description and drawings.

[0042] The present application provides a Chinese multi-label recognition method based on multi-scale fusion and attention mechanism, which uses a convolutional neural network to extract picture information, uses a local attention mechanism to calculate an attention score for feature maps of a third convolutional block and a fourth convolutional block of the convolutional neural network, uses a global attention mechanism for a feature map generated by a fifth convolutional block, splices the obtained results, uses a linear layer for classification, and obtains the final recognition result.

[0043] As shown in Figure 1 The present application constructs a Chinese recognition network model under the pytorch deep learning framework, and the Chinese recognition network model diagram mainly includes a convolutional neural network (picture feature extraction network), an attention module and a classification module.

[0044] With the development of artificial intelligence technology, many classic convolutional neural network models have appeared, but as the number of network layers increases, the precision will reach saturation and then rapidly decline. This degradation is not caused by overfitting, but by the deepening of the network layers, which leads to higher training errors. In order to solve the degradation problem of the convolutional neural network, a residual neural network appears. The residual block is added to the convolutional network, which solves the problem of poor training results caused by gradient disappearance.

[0045] In one embodiment, a ResNet18 network is used as the picture feature extraction network, and the ResNet18 is composed of multiple 3*3 convolutional layers, which can effectively extract picture information, and its specific structure is shown in Table 1.

[0046] Table 1 ResNet18 structure

[0047]

[0048] ResNet18 is mainly composed of five convolutional blocks, the first convolutional block includes a convolutional layer with a convolution kernel size of 7x7, an output channel of 64 and a step of 2, and a maximum pooling operation with a kernel size of 3 and a step of 2.

[0049] The second convolutional block includes four convolutional layers with a kernel size of 3*3, 64 output channels and a step size of 2, and a max-pooling operation with a kernel size of 3 and a step size of 2. However, the result needs to be added to the input of the two convolutional layers after passing through each two convolutional layers.

[0050] The third convolutional block is composed of four convolutional layers with a kernel size of 3*3 and 128 output channels, but the step size of the first convolutional layer is 2 and the step size of the remaining convolutional layers is 1. However, the result needs to be added to the input of the two convolutional layers after passing through each two convolutional layers. In the first addition, in order to ensure the same number of channels, the input needs to be processed using a convolutional layer with an output size of 128 and a kernel size of 1*1.

[0051] The fourth convolutional block is composed of four convolutional layers with a kernel size of 3*3 and 256 output channels, but the step size of the first convolutional layer is 2 and the step size of the remaining convolutional layers is 1. However, the result needs to be added to the input of the two convolutional layers after passing through each two convolutional layers. In the first addition, in order to ensure the same number of channels, the input needs to be processed using a convolutional layer with an output size of 256 and a kernel size of 1*1.

[0052] The fifth convolutional block is composed of four convolutional layers with a kernel size of 3*3 and 512 output channels, but the step size of the first convolutional layer is 2 and the step size of the remaining convolutional layers is 1. However, the result needs to be added to the input of the two convolutional layers after passing through each two convolutional layers. In the first addition, in order to ensure the same number of channels, the input needs to be processed using a convolutional layer with an output size of 512 and a kernel size of 1*1.

[0053] ResNet18 generally only uses the feature map generated by the last layer of convolution for classification and discrimination. According to the principle of convolutional neural network, the last layer of convolution feature map represents the global information of the picture, but when multiple dishes appear in the picture, a dish often only appears in a local area of the picture, that is, belongs to a small target object, and the global information cannot well represent the characteristics of a single dish. In the convolutional neural network, the earlier the position of the convolutional block, the smaller the field of view of the extracted feature map, and the better the local features or small target object features extracted. Therefore, in the present application, the feature maps of the local information of the pictures generated by the third, fourth and fifth convolutional blocks are all utilized.

[0054] The attention module includes a global attention module and two local attention modules.

[0055] (1) Global attention module

[0056] The global attention module is used to calculate the global attention score of the feature map generated by the fifth convolutional block of the convolutional neural network. In the global attention module, the CBAM algorithm is adopted in the present application.

[0057] Attention mechanisms can not only tell the model which features are more important, but also improve the expressiveness of feature maps. As shown in Figures 2-3 CBAM combines spatial and channel attention mechanisms. CBAM is a simple and effective attention module that can be used in feedforward neural networks. After inputting the feature map, CBAM calculates and infers the attention map in the channel dimension and the spatial dimension of the image, multiplies the attention map with the input feature map, and obtains the feature map enhanced by the attention mechanism. Because CBAM is a lightweight general module, it can be integrated into any CNN framework and can be trained end-to-end with basic CNN.

[0058] In CBAM, the model first extracts channel attention features from the feature image. First, the input feature map is respectively subjected to maximum pooling and average pooling operations, and then the maximum pooled result and the average pooled result are respectively subjected to a shared network layer composed of a multilayer perceptron and a hidden layer. The two results are added element by element, and finally activated by a sigmoid function.

[0059] The formula is as follows:

[0060] M C (F) = σ(MLP(AvgPool(F)) + MLP(MaxPool(F)));

[0061] Where F represents the feature image. σ represents the sigmoid operation. MLP is a multilayer perceptron with one hidden layer.

[0062] The next step is to extract spatial feature attention from the feature map that has passed through the channel attention layer. First, apply average pooling and maximum pooling operations along the channel axis, connect them, and then apply a convolution layer to generate a feature attention map.

[0063] The formula can be expressed as follows:

[0064] M s (F) = σ(f 7×7 ([AvgPool(F); MaxPool(F)]);

[0065] Where σ represents the sigmoid operation, and f 7×7 represents a convolution operation with a filter size of 7x7. The overall feature extraction formula of CBAM can be expressed as:

[0066]

[0067]

[0068] wherein, represents element-wise multiplication, F" represents the final output result, F represents the input feature map, i.e. the feature map generated by the fifth convolutional block, and F" represents the output of the picture after passing through the channel attention mechanism. Since in the present model, only the feature map generated by the fifth convolutional block is calculated for global attention, F 5 represents the result of the feature map of the fifth convolutional block after global attention calculation, i.e. using F 5 to represent F".

[0069] After global attention extraction, dimension reduction is also needed because F 5 is 7x7x512. When dimension reduction is performed, the weighted sum of average pooling and maximum pooling is used.

[0070] output 5 = pooling(F 5 ) = λ * Maxpool(F 5 ) + Avgpool(F 5 );

[0071] wherein, output 5 is the result after pooling, and its size is 1x1x512. λ is a weight parameter, which is trainable, and its own gradient will be generated in the back propagation process during the training process, and then updated according to the gradient. The initial value of λ is set to a random number between 0 and 1.

[0072] (2) Local attention module

[0073] The present application adopts two local attention modules to calculate the local attention score of the feature map output by the third convolutional block and the fourth convolutional block, respectively.

[0074] Since each food only appears in a certain part of the picture, the present application adopts a local attention mechanism to make the model pay more attention to local details.

[0075] As Figure 4 shown, in the local attention mechanism, the CBAM algorithm is improved. First, the feature map after passing through the channel attention mechanism is evenly cut into four parts, which are the top left, bottom left, top right and bottom right of the feature map. Then the spatial attention in CBAM is used to calculate the spatial attention, and after the calculation is completed, the result is pooled, and finally the four results are spliced.

[0076] For the feature map output by the third convolutional block and the fourth convolutional block, the pooling results of each part of the feature map are obtained through the following relationship, respectively:

[0077] pooling(F i) = λ * Maxpool (F i ) + Avgpool (F i ) ;

[0078] In the formula, F i represents the feature map after attention calculation of the i-th part of the feature map, Maxpool (F i ) represents the result of the maximum pooling of F i , Avgpool (F i ) represents the result of the average pooling of F i , pooling (F i ) represents the result of the pooling operation of F i , and λ is a weight parameter that will be updated according to the weight generated by back propagation during training. The initial value of λ is set to a random number between 0 and 1.

[0079] After splicing the four part results, the output result of the local attention block is:

[0080] output j = contact [pooling (F0) ; pooling (F1) ; pooling (F2) ; pooling (F3) ] ;

[0081] Wherein, output j , (j = 3, 4) respectively represent the attention features of the output feature maps of the third convolution block and the fourth convolution block; pooling (F0), pooling (F1), pooling (F2), pooling (F3) respectively represent the pooling results of the four parts of the feature map output by the third convolution block (or the fourth convolution block).

[0082] Then, the output results of the two local attention modules and the global attention module are spliced to obtain a splicing vector:

[0083] O = contact [output 3 ; output 4 ; output 5 ] ;

[0084] Wherein, output 3 ; output 4 ; output 5 are the results of attention extraction and pooling of the feature maps output by the third convolution block, the fourth convolution block and the fifth convolution block, respectively.

[0085] The global attention mechanism and the local attention mechanism are simultaneously used in the application. When the local attention mechanism is calculated, the picture is cut, so that the model can pay more attention to local features, and the model can find different features between dishes containing similar ingredients.

[0086] The obtained spliced vector is predicted and classified using a linear layer.

[0087] out = linear (O)

[0088] Wherein, out is a category prediction value with the size of Cx1, and C represents the number of categories. The i-th element in out represents the probability value of the i-th category appearing in the image. Wherein linear can be represented as

[0089] out = OW T +b

[0090] Wherein, W is a weight parameter to be learned during training, and b is a bias.

[0091] The maximum value in the out vector is found and recorded as max (out). The out vector is traversed, and if there is an element value greater than 0.85*max (out), it indicates that the model considers that the picture contains the food category corresponding to the element value. It can be represented as:

[0092] out[result] > 0.85*max (out)

[0093] Wherein, result is the output food category set, and out[result] represents the probability value of the output category.

[0094] The Chinese multi-label recognition method based on multi-scale fusion and attention mechanism provided in the application uses residual blocks to extract features, and uses local attention blocks and global attention blocks to process the extracted features, and then classifies after splicing. The specific recognition process is as follows:

[0095] The first convolutional block: the input picture size is 224x224x3, first use the convolutional layer with the output channel of 64, the convolutional kernel size of 7x7 and the step of 2, then use the batchnorm method to normalize, use the relu function to perform nonlinear transformation, and use the maximum pooling operation with the kernel size of 3 and the step of 2. The output feature map size is 56x56x64.

[0096] The second convolutional block: the input feature map is the output feature map of the first convolutional block, and the size is 56x56x64. First, a convolutional layer with an output channel of 64, a convolution kernel size of 3x3, and a step of 1 is used, then batchnorm normalization is used, and a relu function is used for nonlinear transformation. The above operations are repeated four times, and the final output feature map size is 56x56x64.

[0097] The third convolutional block: the input feature map is the output feature map of the second convolutional block, and the size is 56x56x64. First, a convolutional layer with an output channel of 128, a convolution kernel size of 3x3, and a step of 2 is used, then batchnorm normalization is used, and a relu function is used for nonlinear transformation. Then, a convolutional layer with an output channel of 128, a convolution kernel size of 3x3, and a step of 1 is used, then batchnorm normalization and a relu function are used. The feature map obtained at this time is added to the input feature map after a convolutional layer with a convolution kernel size of 1, a step of 2, and an output channel of 128. Then, two convolutional layers with an output channel of 128, a convolution kernel size of 3x3, and a step of 1 are used, and batchnorm normalization and a relu function are used after each convolution. Finally, the output is obtained. The final feature map size is 28x28x128. At this time, since local attention calculation is performed, the feature map is cut into blocks, and attention calculation is performed on each cut block. After pooling operation is performed on them respectively, a 1x1x128 vector is obtained for each block. After splicing, the final output size is 1x1x512.

[0098] The fourth convolutional block: the input feature map is the output feature map of the third convolutional block, with a size of 28x28x128. First, a convolutional layer with an output channel of 256, a convolution kernel size of 3x3, and a step size of 2 is used, followed by batchnorm normalization and a relu function for nonlinear transformation. Then, a convolutional layer with an output channel of 256, a convolution kernel size of 3x3, and a step size of 1 is used, followed by batchnorm normalization and a relu function. The feature map obtained at this time is added to the input feature map after a convolutional layer with a convolution kernel size of 1, a step size of 2, and an output channel of 128. Then, two convolutional layers with an output channel of 256, a convolution kernel size of 3x3, and a step size of 1 are used, and batchnorm normalization and a relu function are used after each convolution. The final feature map size is 14x14x256. At this time, the model calculates the local attention score of the feature map. Since the local attention calculation is performed by cutting the feature map, the attention mechanism needs to be calculated for each cut block. After pooling operation, a 1x1x256 vector is obtained for each block. After concatenation, the final output size is 1x1x1024.

[0099] The fifth convolutional block: the input feature map is the output feature map of the fourth convolutional block, with a size of 14x14x256. First, a convolutional layer with an output channel of 512, a convolution kernel size of 3x3, and a step size of 2 is used, followed by batchnorm normalization and a relu function for nonlinear transformation. Then, a convolutional layer with an output channel of 512, a convolution kernel size of 3x3, and a step size of 1 is used, followed by batchnorm normalization and a relu function. Then, two convolutional layers with an output channel of 512, a convolution kernel size of 3x3, and a step size of 1 are used, and batchnorm normalization and a relu function are used after each convolution. The feature map obtained at this time is added to the input feature map after a convolutional layer with a convolution kernel size of 1, a step size of 2, and an output channel of 128. The final feature map size is 14x14x256. After using average pooling and maximum pooling operations, two 1x1x256 size vectors are obtained, and the weighted sum is output with a size of 1x1x256.

[0100] Connection: The results of the third, fourth, and fifth convolutional blocks after the attention block are concatenated, and the final size is 1x1x2048.

[0101] Classification: A linear classification layer with an input size of 70 is used to classify the concatenated vector.

[0102] The application splices feature vectors of different scales before classification, so that the model considers the fused features of different scales during classification, guarantees that the local features and global features are spliced together, and makes the recognition more accurate.

[0103] Embodiment

[0104] The data set used in the model training of the embodiment is from the First Hospital of Jilin University, the pictures in the data set are obtained by on-site shooting, including 70 Chinese food categories, a total of 4,000 pictures, of which 1,500 pictures contain only one Chinese food category, 2,500 pictures contain three Chinese food categories, which is equivalent to a set meal. In the pictures containing multiple foods, the combination of foods has no internal relationship. At the same time, the data set contains pictures under various illuminations and shooting angles. Before training, the pictures are adjusted to 224x224 size, and finally randomly divided into training set and test set according to the ratio of 8:2.

[0105] The application belongs to a picture multi-label classification algorithm, and a cross-entropy loss is used as a loss function during training. The cross-entropy can be used to determine the closeness between the actual output and the expected output. The specific operation formula is as follows:

[0106]

[0107] Wherein, n represents the food category, y i represents the label value of the i-th dimension, that is, the probability that the current sample belongs to the i-th category, p i represents the probability that the model predicts that the picture is the i-th category.

[0108] The experimental environment of the embodiment training is windows 11, the GPU used is Nvidia 2080i, and the model training is carried out under the deep learning framework based on pytorch. During model training, the stochastic gradient descent algorithm (SGD) is used for iteration, and during the training process, the learning rate is attenuated by 0.1 times every ten epochs. The total number of epochs used in this experiment is 15, and the loss curve in the training process is as shown in Figure 5 .

[0109] It can be seen that with the increase of the number of iterations, the model always tends to be in a convergent state. After testing on the test set, the accuracy is 96.1%, and there are some cases of not being fully checked on some pictures. In the embodiment, the ResNet18 network is used only, and the ResNet18 of the model of the application is replaced with vgg16, and the model network is tested, and after training the same number of times, the loss values are as shown in Table 2.

[0110] Table 2 Model Comparison

[0111]

[0112] From the table, it can be seen that when ResNet18 does not use attention mechanism, its convergence speed is very slow, and the accuracy is much lower than that of the network with attention mechanism and feature fusion after training the same number of rounds. For the vgg network after adding attention mechanism and feature fusion, its effect is not as good as that of the ResNet18 network with feature fusion and attention mechanism.

[0113] In this embodiment, the same hyperparameters and iteration numbers are used to train vit (vision transformer) and its improved algorithm vit_for_small_dataset, and the training time and parameter size are shown in Table 3.

[0114] In the evaluation index of image multi-label classification, the average mean precision (mAP), average per-class precision (CP), average per-class recall (CR), average per-class (CF1) and overall average precision (OP), overall average recall (OR), overall average (Overall F1, OF1) are often used. The calculation formula is as follows:

[0115]

[0116]

[0117]

[0118]

[0119] CF1=(2*cp*cr) / (cp+cr)

[0120] OF1=(2*op*or) / (op+or)

[0121] Where, N i c represents the total number of pictures whose predicted class is i and whose actual class is also i; represents the total number of pictures whose predicted class is i; represents the total number of pictures containing the i-th food; the number of correctly predicted classes in the s-th picture; represents the number of predicted classes in the s-th picture; The actual label number of the picture s is represented. The indicators of the model of the present application and the vit_base model and the vit_for_small_dataset model are compared, and the results are shown in Table 3.

[0122] Table 3: Indicator comparison

[0123]

[0124]

[0125] As can be seen from Table 3, the classification performance of the model of the present application is improved compared with vit, and performs well in various indicators. Although the loss value and the accuracy of the present model finally differ little from those of the vit for small dataset, the present model is more lightweight, and the parameter quantity is only one fifth of that of the vit for small dataset.

[0126] For the multi-label classification problem of food images, the present application proposes a model based on attention mechanism and multi-scale fusion, which improves ResNet18. The feature map of the predicted picture is generated by using ResNet18, different attention mechanisms are used for the feature maps generated by the last three convolutional blocks, and finally a linear layer is used for class prediction of the picture. Finally, the present application trains and tests on 3000 pictures. Compared with the traditional convolutional neural network, the present model converges faster, has high accuracy, and has better multi-label recognition ability. Compared with the ResNet model, the present model has better effect; compared with the vit model, the present model has faster training speed, lighter structure and smaller parameter quantity.

[0127] Although the embodiments of the present application have been disclosed as above, they are not limited to the application and implementation listed in the specification and embodiments, and can be fully applied to various fields suitable for the present application, and other modifications can be easily realized by those skilled in the art, and therefore the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A Chinese multi-label recognition method based on multi-scale fusion and attention mechanism, characterized in that, The method comprises the following steps: constructing a Chinese food recognition network model, which comprises a convolutional neural network, an attention module and a classification module; wherein the convolutional neural network comprises a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block and a fifth convolutional block connected in sequence; the attention module comprises a global attention module and two local attention modules, which correspond to the third convolutional block and the fourth convolutional block one by one; using a camera device to obtain a Chinese food picture, and extracting information of the Chinese food picture through the convolutional neural network; extracting attention features of feature maps output by the third convolutional block and the fourth convolutional module through the two local attention modules, respectively; and extracting attention features of a feature map output by the fifth convolutional block through the global attention module; splicing the extraction results of the two local attention modules and the extraction result of the global attention module to obtain a spliced vector; classifying the spliced vector through the classification module to obtain a food category in the Chinese food picture; extracting attention features of a feature map output by the third convolutional block through the local attention module, which comprises the following steps: cutting the feature map after channel attention mechanism into four parts averagely; performing attention calculation on the four parts respectively to obtain attention scores of each part; pooling the attention scores of each part, and splicing the pooled results of each part to obtain the attention features of the feature map output by the third convolutional block; the classification module classifying the spliced vector, which comprises the following steps: using a linear layer to perform category prediction on the spliced vector: out = OW T + b; wherein out is a category prediction value with a size of C×1, C represents the number of categories; W is a weight parameter to be learned during training, and b is a bias; the maximum value in the out vector is recorded as max(out), and the out vector is traversed; if there is an element value greater than 0.85*max(out), the Chinese food picture contains the food category corresponding to the element value, which is represented as: out[result]>0.85*max(out); wherein result is a set of output food categories, and out[result] represents the probability value of the output category.

2. The Chinese multi-label recognition method based on multi-scale fusion and attention mechanism according to claim 1, characterized in that, the pooled results of each part of the feature map are obtained through the following relationship: pooling(F i ) = λ * Maxpool(F i ) + Avgpool(F i ); where F i represents the feature map after calculating the attention score of the i-th part, Maxpool(F i ) represents the result of the max-pooling operation on F i , Avgpool(F i ) represents the result of the average-pooling operation on F i , pooling(F i ) represents the result of the pooling operation on F i , and λ represents a weight parameter.

3. The Chinese multi-label recognition method based on multi-scale fusion and attention mechanism according to claim 2, characterized in that, the method for extracting attention features of a feature map output by the fourth convolutional block is the same as the method for extracting attention features of a feature map output by the third convolutional block.

4. The Chinese multi-label recognition method based on multi-scale fusion and attention mechanism according to claim 2 or 3, characterized in that, the size of the picture input by the first convolutional block is 224×224×3.

5. The Chinese multi-label recognition method based on multi-scale fusion and attention mechanism according to claim 4, characterized in that, the third convolutional block comprises four convolutional layers, and the output channel of each convolutional layer is 128, and the convolution kernel size is 3×3.

6. The Chinese multi-label recognition method based on multi-scale fusion and attention mechanism according to claim 5, characterized in that, the fourth convolutional block comprises four convolutional layers, and the output channel of each convolutional layer is 256, and the convolution kernel size is 3×3.

7. The Chinese multi-label recognition method based on multi-scale fusion and attention mechanism according to claim 6, characterized in that, the fifth convolutional block comprises four convolutional layers, and the output channel of each convolutional layer is 512, and the convolution kernel size is 3×3.

8. The Chinese multi-label recognition method based on multi-scale fusion and attention mechanism according to claim 7, characterized in that, the convolutional neural network adopts ResNet18.

Images