A training method for image recognition model and image recognition method

Abstract

The embodiment of the present invention provides a training method for an image recognition model and an image recognition method, the training method comprising: after recording a first image matrix of a sample image, segmenting and shuffling it to obtain a second image matrix; extracting image features and obtaining image classification results respectively through corresponding convolutional neural networks; solving a distillation loss function according to image features, and solving a classification loss function according to image classification results; optimizing the model by optimizing the distillation loss function and the classification loss function, and ending the training when the distillation loss function is less than a preset first threshold and the classification loss function is less than a preset second threshold, thereby obtaining a trained image recognition model. The embodiment of the present invention is conducive to realizing local feature capture and extracting more effective features, without the need for any manual annotation information, and can achieve the same accuracy as strong supervised fine-grained recognition. In terms of the model, it can reduce the time and space consumption of the algorithm and improve robustness.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image recognition model training method and an image recognition method. Background Art

[0002] Fine-grained recognition is also called refined recognition. It is different from existing general image analysis tasks. Fine-grained image recognition requires more detailed types of recognition and finer granularity. It is necessary to distinguish more subdivided subcategories in a large category and distinguish and recognize objects with subtle differences.

[0003] For example, general image classification only needs to distinguish between the two major categories of "bird" and "flower", while fine-grained image classification requires distinguishing the fine-grained subcategories under the "flower" category, that is, distinguishing between "rose" and "rose". Therefore, fine-grained image recognition requires finding the subtle differences between different subcategories of the same category, which greatly increases its difficulty and challenges.

[0004] At present, fine-grained image recognition has a wide range of application scenarios in life and industry. As an image recognition technology, it is an indispensable and important technology in the field of artificial intelligence. At the same time, because it distinguishes more fine-grained granularity, fine-grained image recognition technology can greatly improve existing recognition technology and help improve the accuracy of related upper-level technologies.

[0005] Existing fine-grained classification models can be divided into two categories according to the strength of the supervisory information they use: "classification models based on strong supervisory information" and "classification models based on weak supervisory information".

[0006] Among them, the classification model based on strong supervision information introduces two additional manual annotation information during the training process, namely, the target annotation box and the key part annotation points. For these two additional information, the strongly supervised classification model can detect the foreground object with the help of the target annotation box, eliminating the noise interference caused by the background; and the key part annotation points can be used to locate the key points with significant differences in the target, and the local features of the image can be efficiently extracted at these key points. Therefore, through the positioning provided by these two additional information, the strongly supervised classification model can better extract object information in precise places, eliminate the interference caused by irrelevant information on the background of the picture and other objects, obtain higher accuracy, and achieve better results.

[0007] The classification model based on weak supervision information is the opposite. It does not use any additional manual annotation information, and only relies on pictures and their classification labels to complete the training and learning of the entire algorithm. This type of algorithm does not require a lot of manual input and is more convenient and concise in actual application scenarios. In general, the accuracy of the classification model algorithm based on weak supervision information is not as good as that of the classification model algorithm based on strong supervision information. However, thanks to the development of deep learning in recent years, the classification model algorithm based on weak supervision information has introduced convolutional neural networks for training, and its accuracy has been greatly improved, and it has gradually become a trend in fine-grained image recognition research.

[0008] The key point of fine-grained recognition algorithms is how to dig out the subtle differences in the image, that is, the extraction of local features. Since it is difficult to find discriminative features, the task of fine-grained recognition is very challenging. For weakly supervised fine-grained recognition algorithms, it is impossible to accurately locate the target position and key parts with the help of manual annotation information, and local features can only be extracted based on the image. For a picture, there are many local features extracted. How to eliminate erroneous interference features from many local features and learn useful features is a difficult problem. Existing local feature extraction usually uses an enumeration method to cut out the component area using different step sizes or scales in the whole picture, and then propose features for the component area. However, this method is very time-consuming and easily interfered by background information, extracting a large number of regional features that are useless for recognition. In addition, different lighting conditions and improper shooting angles of the picture will also interfere with weakly supervised fine-grained recognition. In these cases, the accuracy of weakly supervised fine-grained recognition is low and the robustness is poor. Therefore, it is still very challenging for weakly supervised fine-grained recognition to achieve good robustness and high recognition rate. Summary of the invention

[0009] In order to solve the problems in the prior art, an embodiment of the present invention provides an image recognition model training method and an image recognition method.

[0010] In a first aspect, an embodiment of the present invention provides a training method for an image recognition model, comprising: after recording a first image matrix of a sample image, segmenting and shuffling the sample image to obtain a second image matrix of the shuffled sample image; inputting the first image matrix into a first convolutional neural network, extracting first image features through the first convolutional neural network and obtaining a first image classification result; and inputting the second image matrix into a second convolutional neural network, extracting second image features through the second convolutional neural network and obtaining a second image classification result; solving a preset distillation loss function based on the first image features and the second image features, the smaller the distillation loss function, the better the first convolutional neural network. The closer the network is to the second convolutional neural network in feature calculation process; and, according to the first image classification result and the second image classification result, a preset classification loss function is solved, the smaller the classification loss function is, the closer the first convolutional neural network and the second convolutional neural network are to the true value in classification results; the first convolutional neural network and the second convolutional neural network are optimized by continuously optimizing the distillation loss function and the classification loss function, and the training ends when the distillation loss function is less than a preset first threshold and the classification loss function is less than a preset second threshold, so as to obtain a trained image recognition model constructed by the first convolutional neural network and the second convolutional neural network.

[0011] Furthermore, the segmentation and shuffling of the sample image specifically includes: first, dividing the image into multiple image blocks; then, first performing a shuffling operation on the image blocks in the row direction, and then performing a shuffling operation on the image blocks in the column direction; or, first performing a shuffling operation on the image blocks in the column direction, and then performing a shuffling operation on the image blocks in the row direction.

[0012] Furthermore, the shuffling operation of the image blocks in the row direction includes: for each image block in each row, within a preset first step size range, according to the value of a first random variable, exchanging the position in the row direction with the image block at the corresponding position; the shuffling operation of the image blocks in the column direction includes: for each image block in each column, within a preset second step size range, according to the value of a second random variable, exchanging the position in the column direction with the image block at the corresponding position.

[0013] Furthermore, solving the preset distillation loss function based on the first image features and the second image features includes: obtaining a global flow matrix based on the first image features extracted from two adjacent convolutional layers in the first convolutional neural network, and obtaining a local flow matrix based on the second image features extracted from two adjacent convolutional layers in the second convolutional neural network; solving the preset distillation loss function by calculating the L2 norm distance between the global flow matrix and the local flow matrix.

[0014] Furthermore, the expressions of the global flow matrix and the local flow matrix obtained by the image features of two adjacent layers are:

[0015]

[0016] Among them, F 1 ∈R h×w×m Indicates the image features of the upper c1 layer in the two adjacent layers, F 2 ∈R h×w×m It represents the image features of the lower c2 layer in the two adjacent layers, h, w, m represent the height, width and number of channels of the image features respectively, s represents the sequence number of the image height feature, t represents the sequence number of the image width feature, x represents the input image, and W represents the weight parameter of the neural network.

[0017] Furthermore, the expression of the distillation loss function is:

[0018]

[0019] Among them, W global represents the global flow matrix, W local represents the local flow matrix, L flow (W global ,W local ) represents the distillation loss function obtained according to the global flow matrix and the local flow matrix; λ1 represents the weight coefficient; l represents the sequence number of the flow matrix, and the flow matrix includes the global flow matrix and the local flow matrix; n represents the number of the flow matrices for a picture, wherein the number of the global flow matrix and the number of the local flow matrix are the same; x represents the input picture; N represents the number of pictures; Represents the lth global flow matrix of the x picture; Represents the lth local flow matrix of the x picture; Indicates L2 norm distance calculation.

[0020] In a second aspect, an embodiment of the present invention provides an image recognition method based on the above-mentioned image recognition model, including: after recording a first image matrix of an input image, segmenting and shuffling the input image to obtain a second image matrix of the shuffled input image; inputting the first image matrix into the first convolutional neural network, and obtaining a first output vector of a fully connected layer through the first convolutional neural network; and inputting the second image matrix into the second convolutional neural network, and obtaining a second output vector of a fully connected layer through the second convolutional neural network; and obtaining an image recognition result according to the first output vector and the second output vector.

[0021] Furthermore, obtaining the picture recognition result by using the first output vector and the second output vector includes: adding the first output vector and the second output vector to obtain a third output vector, and obtaining the picture recognition result according to the third output vector.

[0022] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method provided in the first aspect or the second aspect when executing the computer program.

[0023] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect or the second aspect.

[0024] The training method of the image recognition model and the image recognition method provided by the embodiments of the present invention respectively input the image matrix of the original image and the image matrix of the scrambled image into two convolutional neural network branches during model training, and comprehensively learn and train the features extracted by the two convolutional neural networks and the classification results, which is conducive to realizing local feature capture and extracting more effective features. It does not require any manual annotation information, and can achieve the same accuracy as strongly supervised fine-grained recognition. In addition, it can reduce the time and space consumption of the algorithm in the model, thereby improving the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 is a flow chart of a training method for an image recognition model provided by an embodiment of the present invention;

[0027] Figure 2 is a flow chart of a training method for an image recognition model provided by another embodiment of the present invention;

[0028] Figure 3 is a flow chart of an image recognition method provided by an embodiment of the present invention;

[0029] Figure 4 is a structural schematic diagram of an image recognition model training device provided by an embodiment of the present invention;

[0030] Figure 5 is a schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention;

[0031] Figure 6 An example physical structure diagram of an electronic device is provided. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Figure 1 FIG. 1 is a flow chart of a training method for an image recognition model provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0034] Step 101: After recording the first image matrix of the sample image, the sample image is segmented and shuffled to obtain a second image matrix of the shuffled sample image.

[0035] The image can be represented by an image matrix, and the elements in the image matrix can be the grayscale values ​​of each pixel. The image recognition model obtained by the training method of the image recognition model provided by the embodiment of the present invention can realize weakly supervised fine-grained image recognition.

[0036] Fine local detail feature representation is the key to fine-grained recognition. This is because local details are more important than global structures for fine-grained recognition, because images from different fine-grained categories usually have the same global structure or shape, but only different local details. Shuffling and reorganizing the image allows the algorithm to discard global structural information and retain local detail information, forcing the model network to focus on distinctive local areas for recognition. The image shuffling step effectively destroys the global structure. At this time, if the classification network wants to recognize these randomly shuffled images, it must find recognizable local areas and learn them. This operation forces the neural network to focus on the details in the image.

[0037] The training method of the image recognition model provided by the embodiment of the present invention combines the original image and the modified image for training. Therefore, before the sample image is shuffled, it is necessary to pre-store the first image matrix of the sample image, which is the image matrix of the sample image before shuffling. Then, the sample image is segmented and shuffled to obtain the second image matrix of the sample image after shuffling, which is the image matrix of the sample image after shuffling.

[0038] Step 102: input the first image matrix into a first convolutional neural network, extract first image features through the first convolutional neural network and obtain a first image classification result; and input the second image matrix into a second convolutional neural network, extract second image features through the second convolutional neural network and obtain a second image classification result.

[0039] In the embodiment of the present invention, a convolutional neural network is used for learning and training, which includes two convolutional neural networks. The input of the first convolutional neural network is the first image matrix of the original image, and the input of the second convolutional neural network is the second image matrix of the scrambled image.

[0040] Therefore, the feature extraction part is divided into two branches, namely global feature extraction and local feature extraction. The basic structure used by these two branches is the same, for example, both can use resnet50 to extract features. The difference is that the local features are obtained by passing the shuffled image φ(I) through the first convolutional neural network, also known as the convolutional neural network f local The global feature is obtained by passing the original image through the second convolutional neural network, which can also be called the convolutional neural network f global The extracted global features (first image features) and local features (second image features) are respectively passed through the fully connected layer to obtain the global feature classification results (first image classification results) and the local feature classification results (second image classification results).

[0041] Step 103: solving a preset distillation loss function according to the first image features and the second image features, the smaller the distillation loss function is, the closer the first convolutional neural network and the second convolutional neural network are in feature calculation process; and, solving a preset classification loss function according to the first image classification result and the second image classification result, the smaller the classification loss function is, the closer the first convolutional neural network and the second convolutional neural network are in classification results to the true value.

[0042] For the two feature streams obtained above (the first image feature and the second image feature), the intermediate features of each layer in the two convolutional neural networks are used to complete the knowledge distillation step. The concept of knowledge distillation (KD) was first proposed by Hinton and is mostly used in convolutional neural networks. The idea is how to perform knowledge conversion technology, that is, extracting knowledge from a perfect teacher neural network to train the student network, so that the student can improve the recognition accuracy while keeping the model parameters small. However, this method has its limitations and it is difficult to optimize a very deep neural network. It is better to teach someone how to fish than to give him fish. The embodiment of the present invention proposes a new knowledge distillation algorithm, which does not directly learn the features of the teacher network, but instead switches to learning the process of calculating the features of the teacher network. It can jump out of the constraints of the depth of the neural network model and achieve better versatility. When facing difficult tasks in computer vision such as fine-grained recognition, it can also well improve the model recognition and performance.

[0043] Therefore, in an embodiment of the present invention, a preset distillation loss function is solved based on the first image feature and the second image feature, and the smaller the distillation loss function is, the closer the first convolutional neural network and the second convolutional neural network are in feature calculation process; and a preset classification loss function is solved based on the first image classification result and the second image classification result, and the smaller the classification loss function is, the closer the first convolutional neural network and the second convolutional neural network are in classification results. The classification loss function can be expressed as the difference between the sum of the output vectors of the first convolutional neural network and the second convolutional neural network and the true value.

[0044] For the input image I and the shuffled image φ(I), they respectively extract the convolutional neural network f through the global feature extraction global and local feature extraction convolutional neural network f local The corresponding global feature output vector C(I) and local feature output vector C(φ(I)) are obtained. Therefore, the classification loss function can be defined as:

[0045]

[0046] Among them, l represents the true classification value of the image, log represents the logarithmic function, Represents a collection of images.

[0047] Step 104: Optimize the first convolutional neural network and the second convolutional neural network by continuously optimizing the distillation loss function and the classification loss function. The training ends when the distillation loss function is less than a preset first threshold and the classification loss function is less than a preset second threshold, thereby obtaining a trained image recognition model constructed by the first convolutional neural network and the second convolutional neural network.

[0048] The smaller the distillation loss function and the classification loss function, the more optimized the model is. By feeding back the neural network, the distillation loss function and the classification loss function are continuously reduced, thereby gradually optimizing the model. The training ends when the distillation loss function is less than a preset first threshold and the classification loss function is less than a preset second threshold, thereby obtaining a trained image recognition model.

[0049] The training method provided by the embodiment of the present invention is generally divided into two parts: a destruction and reorganization part and a knowledge distillation part. The destruction and reorganization part realizes the orderly disruption of the image, destroys the structural information in the image, and ensures that the algorithm extracts more detailed local information; the knowledge distillation part distills and condenses the features extracted from the destroyed image, extracts the most effective features for improving the model recognition rate, and further improves the accuracy of the algorithm. Among them, the knowledge distillation part can include the process of optimizing the model using the distillation loss function and the classification loss function.

[0050] The embodiment of the present invention inputs the image matrix of the original image and the image matrix of the shuffled image into two convolutional neural network branches respectively, and integrates the features extracted by the two convolutional neural networks and the classification results for learning and training, which is conducive to realizing local feature capture and extracting more effective features. It does not require any manual annotation information, and can achieve the same accuracy as strong supervised fine-grained recognition. In addition, it can reduce the time and space consumption of the algorithm in the model, thereby improving the robustness of the system.

[0051] Further, based on the above embodiment, the segmentation and shuffling of the sample image specifically includes: first, dividing the image into multiple image blocks; then, first performing a shuffling operation on the image blocks in the row direction, and then performing a shuffling operation on the image blocks in the column direction; or, first performing a shuffling operation on the image blocks in the column direction, and then performing a shuffling operation on the image blocks in the row direction.

[0052] When the sample image is segmented and shuffled, segmentation is performed first, and then shuffle. When segmenting, the image is divided into multiple image blocks, such as M×N image blocks. After segmentation, the image blocks are shuffled. The image blocks can be shuffled in the row direction first, and then in the column direction; or the image blocks can be shuffled in the column direction first, and then in the row direction.

[0053] On the basis of the above-mentioned embodiment, the embodiment of the present invention improves the flexibility and orderliness of the system by shuffling the image blocks in the row and column directions after the image is segmented.

[0054] Further, based on the above embodiment, the shuffling operation of the image blocks in the row direction includes: for each image block in each row, within a preset first step size range, according to the value of a first random variable, exchanging the position in the row direction with the image block at the corresponding position; the shuffling operation of the image blocks in the column direction includes: for each image block in each column, within a preset second step size range, according to the value of a second random variable, exchanging the position in the column direction with the image block at the corresponding position.

[0055] The destruction and reconstruction idea proposed in the embodiment of the present invention is how to effectively destroy the image so that the structural information of the image is disrupted while highlighting the local information of the image. The sample image is divided into different image blocks, which is actually to divide the first image matrix into different block matrices. As the first step of the algorithm, the core of image scrambling is to scramble the image in an orderly and controllable manner, that is, to permute the block matrix of the image within a controllable range, so as to control the noise introduced by the scrambling operation while highlighting the local features of the image.

[0056] Specifically, the moving step of the image block can be limited. For example, the moving step of the image block in the row direction can be set within the first moving step range. The first moving step can be represented by a first random variable. When each image block moves, the first random variable can have different values, but they are all within the first moving step range. The moving step of the image block in the column direction can be set within a second step range. The second moving step can be represented by a second random variable. When each image block moves, the second random variable can have different values, but they are all within the second moving step range. When each image block moves, the position of the image block at the corresponding position is interchanged.

[0057] Of course, if it is a square image, the image can be divided into N×N blocks, that is, the number of blocks in the row direction and the column direction is the same. When moving, the movement of the image blocks in the row direction and the column direction can also be set to a uniform step size. Taking this as an example, the method of shuffling the image is further explained:

[0058] The image scrambling step can be divided into two sub-operations: segmentation and scrambling. First, the input image is divided into small local blocks, and then a random algorithm is used to scramble them to obtain the scrambled image. The specific operations are as follows:

[0059] For the input image I, first divide the image into N×N sub-regions R i,j , where i and j correspond to the row block number and column block number respectively. The algorithm shuffles the cut sub-regions through the following mechanism: For the region of the jth row, the algorithm first generates a vector q of size N j , whose i-th element q j,i =i+r, where r is a random variable uniformly distributed in (-k, k), and k is an adjustable parameter of the algorithm (1≤k<N), which describes the range of disturbance caused by the scrambling mechanism. Through such a scrambling mechanism, a new sequence can be obtained. The variation range of each element:

[0060]

[0061] Through the above operation, the row shuffle operation of the image can be completed. After the row shuffle, the column shuffle is performed with similar rules, and the following relationship can also be obtained:

[0062]

[0063] After the input image is shuffled in rows and columns, the shuffled image φ(I) is obtained, and the value of its sub-region σ(i,j) can be expressed as:

[0064]

[0065] The image shuffling step effectively destroys the global structure. At this time, if the classification network wants to recognize these randomly shuffled images, it must find recognizable local areas and learn them. This operation forces the neural network to focus on the details in the image, and through the parameter k, ensures that the selection of local areas is jittered in the adjacent areas, thereby controlling the noise introduced by the shuffling operation and highlighting the local features of the image.

[0066] On the basis of the above-mentioned embodiments, the embodiments of the present invention utilize random variables with preset thresholds to shuffle image blocks in row and column directions, and ensure that local areas are jittered within adjacent areas while highlighting local features, thereby controlling the noise introduced by the shuffling operation.

[0067] Further, based on the above embodiment, solving the preset distillation loss function according to the first image features and the second image features includes: obtaining a global flow matrix according to the first image features extracted from two adjacent layers of the convolutional layer in the first convolutional neural network, and obtaining a local flow matrix according to the second image features extracted from two adjacent layers of the convolutional layer in the second convolutional neural network; solving the preset distillation loss function by calculating the L2 norm distance between the global flow matrix and the local flow matrix.

[0068] When solving the preset distillation loss function according to the first image feature and the second image feature, a global flow matrix is ​​obtained according to the first image feature extracted from two adjacent layers of the convolution layer in the first convolutional neural network, and the global flow matrix reflects the change relationship of the features between two adjacent layers of the convolution layer in the first convolutional neural network; a local flow matrix is ​​obtained according to the second image feature extracted from two adjacent layers of the convolution layer in the second convolutional neural network, and the local flow matrix reflects the change relationship of the features between two adjacent layers of the convolution layer in the second convolutional neural network; the preset distillation loss function is solved by calculating the L2 norm distance between the global flow matrix and the local flow matrix. The L2 norm distance indicates the closeness of the feature changes of two adjacent layers of two convolutional neural networks. Therefore, the smaller the L2 norm distance and the smaller the value of the distillation loss function, the closer the feature changes of two adjacent layers of the two convolutional neural networks are.

[0069] The new knowledge distillation algorithm proposed in the embodiment of the present invention, also known as the flow matrix distillation method, calculates the flow matrix between the two networks to find the changing relationship between the features of each layer of the two networks. The two flow matrices are brought close to each other and merged to enable the student network to learn the "solution" of the teacher network to calculate the features, thereby improving the accuracy of fine-grained recognition. In the algorithm flow proposed in the embodiment of the present invention, there is no strict division of roles between the teacher network and the student network. Instead, the effect of knowledge distillation is achieved by bringing the global feature extraction network (the first convolutional neural network) and the local feature extraction network (the second convolutional neural network) close to each other and merging with each other.

[0070] By continuously optimizing the loss function (including the classification loss function of the distillation loss function), the embodiment of the present invention can continuously fuse the global features and local features extracted from the image, and perform mutual fusion, mutual distillation and refinement. Such a process can extract features that are more helpful to the model recognition rate, better improve the accuracy of fine-grained recognition, and can also eliminate the noise introduced by image disruption in this way. At the same time, the flow matrix distillation method has better model generalization by learning the change process of features between the two networks, which can overcome the limitations of knowledge distillation and can be perfectly executed even in the face of very deep neural networks.

[0071] On the basis of the above-mentioned embodiments, the embodiments of the present invention adopt the flow matrix distillation method to learn the changing process of features between two networks so that the model itself has better generalization, can overcome the limitations of knowledge distillation, and can perform perfectly even in the face of very deep neural networks.

[0072] Further, based on the above embodiment, the expressions of the global flow matrix and the local flow matrix obtained by the picture features of two adjacent layers are:

[0073]

[0074] Among them, F 1 ∈R h×w×m Indicates the image features of the upper c1 layer in the two adjacent layers, F 2 ∈R h×w×m It represents the image features of the lower c2 layer in the two adjacent layers, h, w, m represent the height, width and number of channels of the image features respectively, s represents the sequence number of the image height feature, t represents the sequence number of the image width feature, x represents the input image, and W represents the weight parameter of the neural network.

[0075] For a teacher network, the goal is to learn the process of feature changes in its network, that is, the relationship between the features obtained in two adjacent layers in the network. Therefore, the flow matrix G∈R is defined m×n for:

[0076]

[0077] By calculating the flow matrices of the first convolutional neural network and the second convolutional neural network respectively, and continuously optimizing the L2 norm distance between the two, the effect of knowledge distillation can be achieved.

[0078] On the basis of the above embodiments, the embodiments of the present invention improve practicability by providing an expression of a flow matrix.

[0079] Further, based on the above embodiment, the expression of the distillation loss function is:

[0080]

[0081] Among them, W global represents the global flow matrix, W local represents the local flow matrix, L flow (W global ,W local) represents the distillation loss function obtained according to the global flow matrix and the local flow matrix; λ1 represents the weight coefficient; l represents the sequence number of the flow matrix, and the flow matrix includes the global flow matrix and the local flow matrix; n represents the number of the flow matrices for a picture, wherein the number of the global flow matrix and the number of the local flow matrix are the same; x represents the input picture; N represents the number of pictures; Represents the lth global flow matrix of the x picture; Represents the lth local flow matrix of the x picture; Indicates L2 norm distance calculation.

[0082] First, the global flow matrix G of the global feature extraction network is calculated separately global (x; W global ) and the local flow matrix G of the local feature extraction network local (x; W local ), and then calculate the knowledge distillation loss function L flow (W global ,W local ). Since a flow matrix can be calculated based on two layers, there are multiple flow matrices corresponding to one picture. The above distillation loss function is obtained by combining the L2 norm distance of the flow matrices of each picture. In the embodiment of the present invention, it is considered that each flow matrix is ​​equally important, so the same weight coefficient λ1 can be used in the loss function.

[0083] On the basis of the above embodiments, the embodiments of the present invention obtain a distillation loss function by comprehensively analyzing the L2 norm distance of the flow matrices of each image, thereby improving the reliability of the distillation loss function.

[0084] Figure 2 FIG. 1 is a flow chart of a training method for an image recognition model provided by another embodiment of the present invention. Figure 2 As shown, an embodiment of the present invention proposes a training method for an image recognition model based on destruction and reconstruction and knowledge distillation. This method does not require any manual annotation information, can achieve the same accuracy as strong supervised fine-grained recognition, and can reduce the time and space consumption of the algorithm on the model. The method is generally divided into two parts: a destruction and reconstruction part and a knowledge distillation part. The destruction and reconstruction part realizes orderly disruption of the image, destroys the structural information in the image, and ensures that the algorithm extracts more detailed local information; the knowledge distillation part distills and condenses the features extracted from the destroyed image, extracts the most effective features for improving the model recognition rate, and further improves the accuracy of the algorithm.

[0085] First, the algorithm performs an image destruction step to shuffle the image in an orderly manner, that is, to control the disturbance amplitude while shuffling it, so as to effectively control the noise introduced by the shuffle. Through such steps, the original structural information of the image is destroyed, and the algorithm is forced to focus on the local information points in the image, extracting more effective and more accurate local information.

[0086] After the destruction and reorganization part, the algorithm enters the knowledge distillation part, which is completed by two branches. The shuffled image and the original image obtained above are respectively subjected to the convolutional neural network for local feature extraction and global feature extraction, and then the local classification results and global classification results are obtained through the fully connected layer. At the same time, the local flow matrix and global flow matrix required by the algorithm are calculated according to the calculation results of each layer of the convolutional neural network on both sides, and then the knowledge distillation algorithm is used to distill and condense the extracted features, and further obtain the most effective features for improving the recognition rate of the model, and help adjust the parameters of the convolutional neural network, so that the algorithm can integrate global and local features to perform fine-grained classification of images, effectively improving the accuracy of fine-grained recognition.

[0087] Figure 3 is a flow chart of an image recognition method provided by an embodiment of the present invention. The method can use the image recognition model trained by any of the above embodiments to perform image recognition. The method includes:

[0088] Step 201: After recording the first image matrix of the input image, the input image is segmented and shuffled to obtain a second image matrix of the shuffled input image.

[0089] After recording the first image matrix of the input image, the input image can be segmented and shuffled according to the rules of image segmentation and shuffling during model training, thereby obtaining the second image matrix of the shuffled input image. Unlike the sample images during training, the first image matrix in the embodiment of the present invention corresponds to the input image that actually needs to be recognized, and the second image matrix corresponds to the shuffled input image.

[0090] Step 202: input the first image matrix into the first convolutional neural network, and obtain the first output vector of the fully connected layer through the first convolutional neural network; and input the second image matrix into the second convolutional neural network, and obtain the second output vector of the fully connected layer through the second convolutional neural network.

[0091] The first image matrix is ​​input into the first convolutional neural network, and the first output vector of the fully connected layer is obtained through the first convolutional neural network, and the size of each element in the first output vector can represent the probability that the image belongs to the corresponding category. The second image matrix is ​​input into the second convolutional neural network, and the second output vector of the fully connected layer is obtained through the second convolutional neural network, and the size of each element in the second output vector can represent the probability that the image belongs to the corresponding category.

[0092] Step 203: Obtain a picture recognition result according to the first output vector and the second output vector.

[0093] The first output vector and the second output vector may be combined to obtain the image recognition result. For example, the first output vector and the second output vector may be weightedly summed, and the category to which the image belongs may be determined based on the size of the elements in the output vector.

[0094] The embodiment of the present invention performs image recognition by using the image recognition model obtained by the above training method, so as to realize weakly supervised fine-grained image recognition without the need for any manual annotation information, and can achieve the same accuracy as strongly supervised fine-grained recognition.

[0095] Further, based on the above embodiment, obtaining the image recognition result according to the first output vector and the second output vector includes: adding the first output vector and the second output vector to obtain a third output vector, and obtaining the image recognition result according to the third output vector.

[0096] When the image recognition result is obtained according to the first output vector and the second output vector, the first output vector and the second output vector can be directly added to obtain a third output vector, and the category to which the image belongs can be determined according to the size of the elements in the third output vector, thereby obtaining the image recognition result.

[0097] On the basis of the above-mentioned embodiment, the embodiment of the present invention obtains a third output vector by adding the first output vector and the second output vector, and obtains the image recognition result according to the third output vector, thereby improving simplicity.

[0098] Figure 4 Schematic diagram of the structure of an image recognition model training device provided by an embodiment of the present invention. Figure 4As shown, the device includes an image scrambling module 10, a feature extraction and classification module 20, a loss function calculation module 30 and a model optimization module 40, wherein: the image scrambling module 10 is used to: after recording the first image matrix of the sample image, divide and scramble the sample image to obtain the second image matrix of the scrambled sample image; the feature extraction and classification module 20 is used to: input the first image matrix into a first convolutional neural network, extract the first image feature through the first convolutional neural network and obtain the first image classification result; and input the second image matrix into a second convolutional neural network, extract the second image feature through the second convolutional neural network and obtain the second image classification result; the loss function calculation module 30 is used to: solve the preset steam according to the first image feature and the second image feature A distillation loss function, the smaller the distillation loss function is, the closer the first convolutional neural network and the second convolutional neural network are in the feature calculation process; and, according to the first image classification result and the second image classification result, a preset classification loss function is solved, the smaller the classification loss function is, the closer the first convolutional neural network and the second convolutional neural network are in the classification result. The model optimization module 40 is used to: optimize the first convolutional neural network and the second convolutional neural network by continuously optimizing the distillation loss function and the classification loss function, and the training ends when the distillation loss function is less than a preset first threshold and the classification loss function is less than a preset second threshold, so as to obtain a trained image recognition model constructed by the first convolutional neural network and the second convolutional neural network.

[0099] The embodiment of the present invention inputs the image matrix of the original image and the image matrix of the shuffled image into two convolutional neural network branches respectively, and integrates the features extracted by the two convolutional neural networks and the classification results for learning and training, which is conducive to realizing local feature capture and extracting more effective features. It does not require any manual annotation information, and can achieve the same accuracy as strong supervised fine-grained recognition. In addition, it can reduce the time and space consumption of the algorithm in the model, thereby improving the robustness of the system.

[0100] Figure 5 FIG. 1 is a schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention. Figure 5As shown, the device includes an image processing module 100, an output vector acquisition module 200 and an image recognition module 300, wherein: the image processing module 100 is used to: after recording the first image matrix of the input image, divide and shuffle the input image to obtain the second image matrix of the shuffled input image; the output vector acquisition module 200 is used to: input the first image matrix to the first convolutional neural network, and obtain the first output vector of the fully connected layer through the first convolutional neural network; and, input the second image matrix to the second convolutional neural network, and obtain the second output vector of the fully connected layer through the second convolutional neural network; the image recognition module 300 is used to: obtain the image recognition result according to the first output vector and the second output vector.

[0101] The embodiment of the present invention performs image recognition by using the image recognition model obtained by the above training method, so as to realize weakly supervised fine-grained image recognition without the need for any manual annotation information, and can achieve the same accuracy as strongly supervised fine-grained recognition.

[0102] The device provided in the embodiment of the present invention is used for the above method. The specific functions can refer to the above method flow and will not be repeated here.

[0103] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the training method of the image recognition model, the method comprising: after recording the first image matrix of the sample image, dividing and shuffling the sample image, so as to obtain the second image matrix of the shuffled sample image; inputting the first image matrix into the first convolutional neural network, extracting the first image features through the first convolutional neural network and obtaining the first image classification result; and inputting the second image matrix into the second convolutional neural network, extracting the second image features through the second convolutional neural network and obtaining the second image classification result; solving the preset distillation loss function according to the first image features and the second image features, the smaller the distillation loss function is, the better the image classification result is. The closer the first convolutional neural network and the second convolutional neural network are in feature calculation process; and, solving a preset classification loss function according to the first image classification result and the second image classification result, the smaller the classification loss function is, the closer the first convolutional neural network and the second convolutional neural network are to the true value in classification results; the first convolutional neural network and the second convolutional neural network are optimized by continuously optimizing the distillation loss function and the classification loss function, and the training ends when the distillation loss function is less than a preset first threshold and the classification loss function is less than a preset second threshold, thereby obtaining a trained image recognition model constructed by the first convolutional neural network and the second convolutional neural network. Alternatively, the processor 610 may call logic instructions in the memory 630 to execute an image recognition method, which includes: after recording a first image matrix of an input image, segmenting and shuffling the input image to obtain a second image matrix of the shuffled input image; inputting the first image matrix into the first convolutional neural network, and obtaining a first output vector of a fully connected layer through the first convolutional neural network; and inputting the second image matrix into the second convolutional neural network, and obtaining a second output vector of a fully connected layer through the second convolutional neural network; and obtaining an image recognition result based on the first output vector and the second output vector.

[0104] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0105] On the other hand, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the training method of the image recognition model provided by the above-mentioned method embodiments, the method including: after recording a first image matrix of a sample image, segmenting and shuffling the sample image to obtain a second image matrix of the shuffled sample image; inputting the first image matrix into a first convolutional neural network, extracting first image features through the first convolutional neural network and obtaining a first image classification result; and inputting the second image matrix into a second convolutional neural network, extracting second image features through the second convolutional neural network and obtaining a second image classification result; according to the The first image feature and the second image feature are used to solve a preset distillation loss function, and the smaller the distillation loss function is, the closer the first convolutional neural network and the second convolutional neural network are in the feature calculation process; and, according to the first image classification result and the second image classification result, a preset classification loss function is solved, and the smaller the classification loss function is, the closer the first convolutional neural network and the second convolutional neural network are in the classification result. The first convolutional neural network and the second convolutional neural network are optimized by continuously optimizing the distillation loss function and the classification loss function, and the training ends when the distillation loss function is less than a preset first threshold and the classification loss function is less than a preset second threshold, thereby obtaining a trained image recognition model constructed by the first convolutional neural network and the second convolutional neural network. Or, when the program instructions are executed by a computer, the computer can execute the image recognition method provided by the above-mentioned method embodiments, which method includes: after recording the first image matrix of the input picture, dividing and shuffling the input picture to obtain the second image matrix of the shuffled input picture; inputting the first image matrix into the first convolutional neural network, and obtaining the first output vector of the fully connected layer through the first convolutional neural network; and inputting the second image matrix into the second convolutional neural network, and obtaining the second output vector of the fully connected layer through the second convolutional neural network; and obtaining the picture recognition result according to the first output vector and the second output vector.

[0106] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training method of the image recognition model provided in the above-mentioned embodiments, the method comprising: after recording a first image matrix of a sample image, segmenting and shuffling the sample image to obtain a second image matrix of the shuffled sample image; inputting the first image matrix into a first convolutional neural network, extracting first image features through the first convolutional neural network and obtaining a first image classification result; and inputting the second image matrix into a second convolutional neural network, extracting second image features through the second convolutional neural network and obtaining a second image classification result; solving a preset A distillation loss function is provided, the smaller the distillation loss function is, the closer the first convolutional neural network and the second convolutional neural network are in feature calculation process; and, according to the first image classification result and the second image classification result, a preset classification loss function is solved, the smaller the classification loss function is, the closer the first convolutional neural network and the second convolutional neural network are in classification results. The first convolutional neural network and the second convolutional neural network are optimized by continuously optimizing the distillation loss function and the classification loss function, and the training ends when the distillation loss function is less than a preset first threshold and the classification loss function is less than a preset second threshold, thereby obtaining a trained image recognition model constructed by the first convolutional neural network and the second convolutional neural network. Alternatively, when the computer program is executed by a processor, it is implemented to execute the image recognition method provided by the above-mentioned embodiments, and the method includes: after recording the first image matrix of the input picture, the input picture is segmented and shuffled to obtain the second image matrix of the shuffled input picture; the first image matrix is ​​input to the first convolutional neural network, and the first output vector of the fully connected layer is obtained through the first convolutional neural network; and the second image matrix is ​​input to the second convolutional neural network, and the second output vector of the fully connected layer is obtained through the second convolutional neural network; and the picture recognition result is obtained according to the first output vector and the second output vector.

[0107] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for an image recognition model, characterized in that: include: After recording the first image matrix of the sample image, the sample image is segmented and shuffled, thereby obtaining a second image matrix of the shuffled sample image; Inputting the first image matrix into a first convolutional neural network, extracting first image features through the first convolutional neural network and obtaining a first image classification result; and inputting the second image matrix into a second convolutional neural network, extracting second image features through the second convolutional neural network and obtaining a second image classification result; Solving a preset distillation loss function according to the first image feature and the second image feature, the smaller the distillation loss function is, the closer the first convolutional neural network and the second convolutional neural network are in feature calculation process; and solving a preset classification loss function according to the first image classification result and the second image classification result, the smaller the classification loss function is, the closer the first convolutional neural network and the second convolutional neural network are in classification results to the true value; The first convolutional neural network and the second convolutional neural network are optimized by continuously optimizing the distillation loss function and the classification loss function, and the training ends when the distillation loss function is less than a preset first threshold and the classification loss function is less than a preset second threshold, thereby obtaining a trained image recognition model constructed by the first convolutional neural network and the second convolutional neural network; The solving a preset distillation loss function according to the first picture feature and the second picture feature includes: Acquire a global flow matrix based on the first image features extracted from two adjacent layers of the convolutional layer in the first convolutional neural network, and acquire a local flow matrix based on the second image features extracted from two adjacent layers of the convolutional layer in the second convolutional neural network; wherein the global flow matrix reflects the change relationship of features between two adjacent layers of the convolutional layer in the first convolutional neural network, and the local flow matrix reflects the change relationship of features between two adjacent layers of the convolutional layer in the second convolutional neural network; By calculating the L2 norm distance between the global flow matrix and the local flow matrix, the preset distillation loss function is solved.

2. The method for training an image recognition model according to claim 1, characterized in that: The segmenting and shuffling of the sample images specifically includes: First, the image is divided into a plurality of image blocks; then, the image blocks are shuffled in the row direction first, and then in the column direction; or, the image blocks are shuffled in the column direction first, and then in the row direction.

3. The method for training an image recognition model according to claim 2, characterized in that: The shuffling operation of the image blocks in the row direction includes: for each image block in each row, exchanging the position of the image block in the row direction with the image block at the corresponding position according to the value of the first random variable within a preset first step length range; The shuffling operation of the image blocks in the column direction includes: for each image block in each column, exchanging the position of the image block in the column direction with the image block at the corresponding position according to the value of the second random variable within a preset second step range.

4. The method for training an image recognition model according to claim 1, characterized in that: The expressions of the global flow matrix and the local flow matrix obtained by the image features of two adjacent layers are: Among them, F 1 ∈R h×w×m Indicates the image features of the upper c1 layer in the two adjacent layers, F 2 ∈R h×w×m It represents the image features of the lower c2 layer in the two adjacent layers, h, w, m represent the height, width and number of channels of the image features respectively, s represents the sequence number of the image height feature, t represents the sequence number of the image width feature, x represents the input image, and W represents the weight parameter of the neural network.

5. The method for training an image recognition model according to claim 4, characterized in that: The expression of the distillation loss function is: Among them, W global represents the global flow matrix, W local represents the local flow matrix, L flow (W global ,W local ) represents the distillation loss function obtained according to the global flow matrix and the local flow matrix; λ1 represents the weight coefficient; l represents the sequence number of the flow matrix, and the flow matrix includes the global flow matrix and the local flow matrix; n represents the number of the flow matrices for a picture, wherein the number of the global flow matrix and the number of the local flow matrix are the same; x represents the input picture; N represents the number of pictures; Represents the lth global flow matrix of the x picture; Represents the lth local flow matrix of the x picture; Indicates L2 norm distance calculation.

6. An image recognition method based on the image recognition model according to any one of claims 1 to 5, characterized in that: include: After recording the first image matrix of the input image, the input image is segmented and shuffled, thereby obtaining a second image matrix of the shuffled input image; Inputting the first image matrix into the first convolutional neural network, and obtaining a first output vector of a fully connected layer through the first convolutional neural network; and inputting the second image matrix into the second convolutional neural network, and obtaining a second output vector of the fully connected layer through the second convolutional neural network; A picture recognition result is obtained according to the first output vector and the second output vector.

7. The image recognition method according to claim 6, characterized in that: The obtaining of the picture recognition result according to the first output vector and the second output vector comprises: The first output vector and the second output vector are added to obtain a third output vector, and the picture recognition result is obtained according to the third output vector.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements the steps of the training method of the image recognition model as described in any one of claims 1 to 5 or the steps of the image recognition method as described in any one of claims 6 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the training method of the image recognition model as described in any one of claims 1 to 5 or the steps of the image recognition method as described in any one of claims 6 to 7.

Images