Image classification method, image classification model training method, device and equipment

Abstract

The invention provides an image classification method, an image classification model training method, an image classification model training device and image classification model training equipment. The image classification model training method comprises the steps of acquiring a feature extraction layer of a to-be-trained image classification model; a training data set is generated according to N image sample sets, N is an integer greater than or equal to 2, each image sample set comprises a plurality of image samples, the image samples belonging to the same image sample set belong to the same category, and the image samples belonging to different image sample sets belong to different categories; the training data set comprises a plurality of positive sample pairs and a plurality of negative sample pairs, two image samples in the positive sample pairs belong to the same image sample set, and two image samples in the negative sample pairs belong to different image sample sets; and training the feature extraction layer by using the training data set to obtain a trained feature extraction layer. According to the method, the image classification model with relatively high accuracy can be trained even if small-sample images are used.

Description

Image classification method, image classification model training method, device and equipment Technical Field

[0001] The present invention relates to the field of image classification technology, and more particularly to an image classification method, an image classification model training method, an apparatus, and equipment. Background Art

[0002] Training image classification models often requires a large number of image samples to achieve high accuracy. However, in reality, the number of image samples available for many scenes is very small, and there is a certain amount of noise (for example, the image samples contain a large amount of background), which makes it difficult to achieve comprehensible results when training image classification models.

[0003] Summary of the Invention

[0004] Embodiments of the present invention provide an image classification method, an image classification model training method, an apparatus, and equipment, which are used to solve the problem that when there are few image samples used to train the image classification model, it is difficult to achieve an understandable effect in the training of the image classification model.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides an image classification method, comprising:

[0007] Get the image to be classified;

[0008] An image classification model is used to classify the image to be classified, wherein the image classification model includes a feature extraction layer, and the feature extraction layer is trained using a training data set generated by N image sample sets, where N is an integer greater than or equal to 2, and each of the image sample sets includes multiple image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories, and the training data set includes multiple positive sample pairs and multiple negative sample pairs, the two image samples in the positive sample pair belong to the same image sample set, and the two image samples in the negative sample pair belong to different image sample sets.

[0009] In a second aspect, an embodiment of the present invention provides a method for training an image classification model, comprising:

[0010] Get the feature extraction layer of the image classification model to be trained;

[0011] Generating a training data set based on N image sample sets, where N is an integer greater than or equal to 2, each of the image sample sets includes multiple image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories, and the training data set includes multiple positive sample pairs and multiple negative sample pairs, the two image samples in the positive sample pairs belong to the same image sample set, and the two image samples in the negative sample pairs belong to different image sample sets;

[0012] The feature extraction layer is trained using the training data set to obtain a trained feature extraction layer.

[0013] Optionally, obtaining a feature extraction layer of an image classification model to be trained includes:

[0014] The feature extraction layer is obtained by removing several layers behind the pre-trained image classification model.

[0015] Optionally, in different image samples in the same image sample set, main objects belonging to the same category are located in different positions.

[0016] Optionally, generating a training data set according to N image sample sets includes:

[0017] Generate positive sample pairs, where the number of all positive sample pairs is T is an integer greater than or equal to 0 and less than or equal to N-1, is the number of positive sample pairs corresponding to the T-th image sample set, K T is the number of image samples in the T-th image sample set;

[0018] Generate negative sample pairs, where the number of all negative sample pairs is K V is the number of image samples in the Vth image sample set;

[0019] The training data set is formed according to the positive sample pairs and the negative sample pairs.

[0020] Optionally, forming the training data set according to the positive sample pairs and the negative sample pairs includes:

[0021] Determine the batch_size B and the number of iterations I used for training, where batch_size is the total number of positive and negative sample pairs used in each training;

[0022] The serial numbers of all positive sample pairs are Randomly arrange them and combine them in order to get no less than The sequence number set Data of positive sample pairs + ;

[0023] The serial numbers of all negative sample pairs are Randomly arrange them and combine them in order to get no less than The sequence number set Data of negative sample pairs - ;

[0024] According to the sequence number set Data + and the sequence number set Data - , forming the training data set.

[0025] Optionally, the training the feature extraction layer using the training data set includes:

[0026] Alternately from the sequence number set Data + and the sequence number set Data - Positive sample pairs and negative sample pairs are selected to train the feature extraction layer.

[0027] Optionally, the adopting the training data set to train the feature extraction layer to obtain a trained feature extraction layer includes:

[0028] For each sample pair in the training data set, the feature extraction layer is used to extract feature maps F0 and F1 of the two image samples in the sample pair respectively; the sample pair includes a positive sample pair and a negative sample pair;

[0029] Determine the position of the feature block with the highest similarity in the feature maps F0 and F1 as the best matching position;

[0030] Determining an error between the similarity of the best matching position and the true value of the sample pair, determining a loss value based on the error, and adjusting parameters of the feature extraction layer based on the loss value;

[0031] If the accuracy of the feature extraction layer reaches a threshold or the loss value no longer decreases or the number of iterations reaches a preset value, the training is terminated.

[0032] Optionally, determining the position of the feature block with the highest similarity in the feature maps F0 and F1 as the best matching position includes:

[0033] Determine the size and scanning step of the matching feature block, where the side length of the matching feature block is H is the feature height output by the feature extraction layer, W is the feature width output by the feature extraction layer, and R is the ratio of the area of ​​the main object in the image sample to the entire image area;

[0034] According to the matching feature block and the scanning step size, each feature block of size C*BS*BS is traversed in the feature maps F0 and F1 from the two dimensions of feature width and feature height, and the similarity of the feature blocks in the feature maps F0 and F1 is determined, and the position of the feature block with the highest similarity in the feature maps F0 and F1 is selected as the best matching position, where C is the number of channels output by the feature extraction layer.

[0035] Optionally, after obtaining the trained feature extraction layer, the following steps are further included:

[0036] The N image sample sets are used to train the layers of the image classification model to be trained except the trained feature extraction layer to obtain a trained image classification model.

[0037] In a third aspect, an embodiment of the present invention provides an image classification device, comprising:

[0038] An acquisition module, used to acquire images to be classified;

[0039] An image classification module is configured to classify the image to be classified using an image classification model, wherein N is an integer greater than or equal to 2, the image classification model includes a feature extraction layer, and the feature extraction layer is trained using a training data set generated by N image sample sets. Each of the image sample sets includes multiple image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories. The training data set includes multiple positive sample pairs and multiple negative sample pairs, the two image samples in the positive sample pair belong to the same image sample set, and the two image samples in the negative sample pair belong to different image sample sets.

[0040] In a fourth aspect, an embodiment of the present invention provides a training device for an image classification model, comprising:

[0041] An acquisition module, used to obtain the feature extraction layer of the image classification model to be trained;

[0042] a generating module, configured to generate a training data set based on N image sample sets, where N is an integer greater than or equal to 2, each of the image sample sets includes a plurality of image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories, the training data set includes a plurality of positive sample pairs and a plurality of negative sample pairs, the two image samples in the positive sample pairs belong to the same image sample set, and the two image samples in the negative sample pairs belong to different image sample sets;

[0043] The first training module is used to train the feature extraction layer using the training data set to obtain a trained feature extraction layer.

[0044] In a fifth aspect, an embodiment of the present invention provides an electronic device comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the image classification method according to the first aspect are implemented, or when the program is executed by the processor, the steps of the image classification model training method according to the second aspect are implemented.

[0045] In a sixth aspect, an embodiment of the present invention provides a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the steps of the image classification method described in the first aspect above; or, when the computer program is executed by a processor, the computer program implements the steps of the image classification model training method described in the second aspect above.

[0046] In an embodiment of the present invention, in the process of training the image classification model, the feature extraction layer of the image classification model is first obtained, and the feature extraction layer is trained using image samples, so that the trained feature extraction layer can effectively extract the features of the main object in the image sample, which is used for the training of other layers of the image classification model, thereby eliminating the influence of noise in the image sample on the training of the image classification model, and even using small image samples, an image classification model with higher accuracy can be trained. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0048] FIG1 is a flow chart of a method for training an image classification model according to an embodiment of the present invention;

[0049] FIG2 is a schematic diagram of an image sample set according to an embodiment of the present invention;

[0050] FIG3 is a schematic diagram of a feature block matching process according to an embodiment of the present invention;

[0051] FIG4 is a schematic diagram of a flow chart of an image classification method according to an embodiment of the present invention;

[0052] FIG5 is a schematic structural diagram of an image classification device according to an embodiment of the present invention;

[0053] FIG6 is a schematic diagram of the structure of a training device for an image classification model according to an embodiment of the present invention;

[0054] FIG7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] Referring to FIG1 , an embodiment of the present invention provides a method for training an image classification model, including:

[0057] Step 11: Obtain the feature extraction layer of the image classification model to be trained;

[0058] Optionally, obtaining the feature extraction layer of the image classification model to be trained includes: removing several layers behind the pre-trained image classification model to obtain the feature extraction layer.

[0059] Furthermore, the feature extraction layer of the image classification model to be trained can be obtained by:

[0060] The first method is to download a pre-trained image classification model from an open source project, extract the feature extraction layer of the image classification model, and remove the last several layers (such as the last 6 layers of the ResNet101 model).

[0061] The second method is to use a public dataset (such as ImageNet) to train an image classification model (such as ResNet101), extract the feature extraction layer of the trained image classification model, and remove the last several layers (such as the last 6 layers of the ResNet101 model).

[0062] Of course, other methods may also be used to obtain the image classification model to be trained and the feature extraction layer in the image classification model, and are not limited to the above two methods.

[0063] In an embodiment of the present invention, the dimension of the feature map output by the feature extraction layer is F = {C, H, W}, where C is the number of channels, H is the feature height, and W is the feature width. For example, the feature map output by the ResNet101 model after removing the last 6 layers has a C of 64, an H of 56, and a W of 56.

[0064] Step 12: generating a training data set based on N image sample sets, where N is an integer greater than or equal to 2, each of the image sample sets includes multiple image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories, the training data set includes multiple positive sample pairs and multiple negative sample pairs, the two image samples in the positive sample pairs belong to the same image sample set, and the two image samples in the negative sample pairs belong to different image sample sets;

[0065] Step 13: Use the training data set to train the feature extraction layer to obtain a trained feature extraction layer.

[0066] In an embodiment of the present invention, in the process of training the image classification model, the feature extraction layer of the image classification model is first obtained, and the feature extraction layer is trained using image samples, so that the trained feature extraction layer can effectively extract the features of the main object in the image sample, which is used for the training of other layers of the image classification model, thereby eliminating the influence of noise in the image sample on the training of the image classification model, and even using small image samples, an image classification model with higher accuracy can be trained.

[0067] In an embodiment of the present invention, optionally, in different image samples in the same image sample set, the main objects belonging to the same category are in different positions. Please refer to Figure 2. Figure 2(a) shows two image samples in the image sample set of the category of dog. It can be seen from Figure 2(a) that the image sample below contains a large amount of background, and the positions of the main objects (dogs) in the two image samples are also different. Figure 2(b) shows two image samples in the image sample set of the category of hamburgers. It can be seen from Figure 2(b) that both image samples contain a large amount of background, and the positions of the main objects (hamburgers) in the two image samples are also different. Figure 2(c) shows the feature maps of the two image sample images extracted by the feature extraction layer. It can be seen that the features are not in the same position. The above situation will lead to difficulties in learning the image classification model.

[0068] In an embodiment of the present invention, the features of the main object in the image can be extracted through the trained feature extraction layer, and the influence of background noise can be removed. Then, the image classification model can be trained based on the features of the main object, thereby improving the efficiency and accuracy of the image classification model training. Even with a small sample, a high-precision image classification model can be trained.

[0069] In the embodiment of the present invention, optionally, generating a training data set according to N image sample sets includes:

[0070] Step 121: Generate positive sample pairs, where the number of all positive sample pairs is T is an integer greater than or equal to 0 and less than or equal to N-1, is the number of positive sample pairs corresponding to the T-th image sample set, K T is the number of image samples in the T-th image sample set;

[0071] The two samples in a positive sample pair come from the same image sample set.

[0072] It is from K T The number of permutations and combinations of two image samples selected from the image samples.

[0073] In the embodiment of the present invention, all positive sample pairs generated can be numbered and recorded. For example, the two image samples in the xth positive sample pair are numbered L. x,0 and L x,1 , then mark the sample pair as Pos x ={L x,0 , L x,1 , 1}, where 1 represents the ground truth, which is the learning goal.

[0074] Step 122: Generate negative sample pairs, where the number of all negative sample pairs is K V is the number of image samples in the Vth image sample set;

[0075] The two samples in a negative sample pair come from different image sample sets.

[0076] Assume that the T-th image sample set has K T image samples, each image sample in the Tth image sample set and each image sample in all other image sample sets (such as the Vth (V≠T)) constitute a negative sample pair. Therefore, the number of negative sample pairs formed by the Tth image sample set is: K T *∑ V≠T K V , K V is the number of image samples in the Vth image sample set, so the number of all negative sample pairs is

[0077] In the embodiment of the present invention, all the negative sample pairs generated can be numbered and recorded. For example, the two image samples in the yth negative sample pair are numbered L. y,0 and L y,1 , then mark the sample pair as Neg y ={L y,0 , L y,1 , 0}, where 0 represents the ground truth, which is the learning target.

[0078] Step 123: Form the training data set according to the positive sample pairs and the negative sample pairs.

[0079] In the embodiment of the present invention, before generating positive sample pairs and negative sample pairs, all image samples can be numbered. The total number of image samples is So the image sample number L can be from 0 to N total -1.

[0080] In the embodiment of the present invention, optionally, forming the training data set according to the positive sample pairs and the negative sample pairs includes:

[0081] Step 1221: Determine the batch_size B and the number of iterations I used for training, where batch_size is the total number of positive sample pairs and negative sample pairs used in each training;

[0082] In an embodiment of the present invention, when training the feature extraction layer, multiple rounds of training are performed, wherein the number of sample pairs used in each training is B, that is, each training uses B sample pairs (including positive samples and negative samples) for parameter optimization, and the B sample pairs jointly determine the direction of the gradient of this training.

[0083] In an embodiment of the present invention, optionally, B is an even number, wherein the number of positive samples and negative samples is B / 2 respectively, that is, the number of positive samples and negative samples used in each training is the same, thereby effectively improving the accuracy of the trained feature extraction layer.

[0084] Of course, in some other embodiments of the present invention, the number of positive samples and negative samples used in training may also be different and can be set as needed.

[0085] In an embodiment of the present invention, optionally, the batch_size B and the number of iterations I used for training can be set based on experience and / or the performance of the electronic device used to perform the training (such as graphics card capacity and / or memory size, etc.).

[0086] Step 1222: Sequence the numbers of all positive sample pairs Randomly arrange them and combine them in order to get no less than The sequence number set Data of positive sample pairs + ;

[0087] in, Is the ceiling symbol.

[0088] In the embodiment of the present invention, in the entire training process, a total of B*I sample pairs (including positive samples and negative samples) are required. Assuming that the positive sample pairs and the negative sample pairs each account for half, the number of positive sample pairs and negative sample pairs required is

[0089] In the embodiment of the present invention, the positive sample pairs are Randomly arrange and combine them in order to get the sequence number set Data of positive sample pairs + In this way, the positive sample pairs used for training are more comprehensive.

[0090] Step 1223: Sequence the numbers of all negative sample pairs Randomly arrange them and combine them in order to get no less than The sequence number set Data of negative sample pairs - ;

[0091] In the embodiment of the present invention, the negative samples are The negative sample pairs are randomly arranged and combined in order to obtain the sequence number set Data of the negative sample pairs. In this way, the negative sample pairs used for training are more comprehensive.

[0092] Step 1224: According to the sequence number set Data + and the sequence number set Data - , forming the training data set.

[0093] In the embodiment of the present invention, optionally, the training of the feature extraction layer using the training data set includes: alternately selecting from the sequence number set Data + and the sequence number set Data - Positive sample pairs and negative sample pairs are selected from the training data to train the feature extraction layer. During the training process, the method of alternately selecting positive sample pairs and negative sample pairs can make the trained feature extraction layer more accurate.

[0094] In the embodiment of the present invention, optionally, the step of training the feature extraction layer using the training data set to obtain a trained feature extraction layer includes:

[0095] For each sample pair in the training data set, the feature extraction layer is used to extract feature maps F0 and F1 of the two image samples in the sample pair respectively; the sample pair includes a positive sample pair and a negative sample pair;

[0096] Determine the position of the feature block with the highest similarity in the feature maps F0 and F1 as the best matching position;

[0097] Determining an error between the similarity of the best matching position and the true value of the sample pair, determining a loss value based on the error, and adjusting parameters of the feature extraction layer based on the loss value;

[0098] If the accuracy of the feature extraction layer reaches a threshold or the loss value no longer decreases or the number of iterations reaches a preset value, the training is terminated.

[0099] In some embodiments, optionally, the step of training the feature extraction layer using the training data set to obtain a trained feature extraction layer includes:

[0100] Step 131: Determine the size and scanning step of the matching feature block, wherein the side length of the matching feature block is H is the feature height output by the feature extraction layer, W is the feature width output by the feature extraction layer, and R is the ratio of the area of ​​the main object in the image sample to the entire image area;

[0101] Since the ratio of the area of ​​the main object to the area of ​​the entire image in different image samples may be different, in the embodiment of the present invention, R may be an estimated value and may be set based on experience.

[0102] In the embodiment of the present invention, optionally, the size of the scanning step is related to the size of the matching feature block. For example, the size of the matching feature block is determined according to the size of the main object.

[0103] For example, assuming that the length and width of the main object are approximately 1 / 2 of the image size, that is, R = 0.25, assuming H = W = 56, BS = 28, the size of the matching feature block can be C*28*28, and the scanning step size is related to BS, for example, it can be set to

[0104] Step 132: For the sample pairs (including positive sample pairs and negative sample pairs) in the training data set, extract feature maps F0 and F1 using the feature extraction layer;

[0105] In this embodiment, the sample pair can be recorded as {L0, L1, G T}(G T =0 or 1, G T is the true value, 0 indicates a negative sample pair, and 1 indicates a positive sample pair). The feature extraction layer can be used to extract the feature maps F0 and F1 of L0 and L1 respectively.

[0106] Step 133: Based on the matching feature blocks and the scanning step size, traverse each feature block of size C*BS*BS in the feature maps F0 and F1 from the feature width and feature height dimensions respectively, and determine the similarity of the feature blocks in the feature maps F0 and F1, and select the position of the feature block with the highest similarity in the feature maps F0 and F1 as the best matching position, where C is the number of channels output by the feature extraction layer;

[0107] Refer to Figure 3. Based on the matching feature blocks and scan step size, we traverse feature map F0 and feature map F1, calculate the similarity between each feature block in feature map F0 (the size of the feature block is the same as the matching feature block) and each feature block in feature map F1, and select the position of the feature block with the highest similarity as the best matching position. The arrows in Figure 3 indicate that the scan direction is from top to bottom and from left to right.

[0108] In the embodiment of the present invention, by traversing the feature graphs through matching feature blocks of appropriate sizes, it is possible to quickly and accurately match the feature blocks with the highest similarity in the two feature graphs.

[0109] In the embodiment of the present invention, optionally, the similarity may be cosine similarity.

[0110] In the embodiment of the present invention, assuming that the coordinates of the upper left corner of the current feature block in the feature map F0 are (x0, y0), the feature block can be recorded as Similarly, the current feature block in the feature map F1 is recorded as (x1, y1) is the coordinate of the upper left corner of the current feature block in the feature map F1. and Expanded into a vector of C*BS*BS dimensions and Calculate the similarity between the current feature block in feature map F0 and the current feature block in feature map F1 Where CS is the cosine similarity; after traversing all positions, the one with the highest similarity is selected as the best matching position (Loc0, Loc1), that is:

[0111] Among them, argmax is a function that finds the parameter (set) of a function. is the coordinate of the upper left corner of the best matching feature block in the feature map F0, is the coordinate of the upper left corner of the best matching feature block in feature map F1.

[0112] Step 134: determining an error between the similarity of the best matching position and the true value of the sample pair, determining a loss value based on the error, and adjusting parameters of the feature extraction layer based on the loss value;

[0113] That is, the calculation error |GT -sim(Loc0,Loc1)|. After calculating the error, the error is input into the loss function to calculate the loss value. In the embodiment of the present invention, a cross entropy loss function can be used as the loss function.

[0114] Step 135: If the accuracy of the feature extraction layer reaches a threshold or the loss value no longer decreases or the number of iterations reaches a preset value (i.e., I above), the training is terminated.

[0115] In an embodiment of the present invention, optionally, after obtaining the trained feature extraction layer, the method further includes: using the N image sample sets to train the layers of the image classification model to be trained except for the trained feature extraction layer to obtain the trained image classification model. That is, the trained feature extraction layer can be merged with the other layers deleted previously to form a complete image classification model, and then the feature extraction layer is frozen (i.e., the feature extraction layer is no longer trained), and the other layers are trained to obtain the final image classification model. Since, during the training process, the feature extraction layer is first used to extract the feature map of the image sample, the noise (such as background) in the image sample can be effectively filtered out, and most of the feature map is content related to the main object, and then the extracted feature map is used to train the other layers, the training efficiency can be effectively improved, and a high-precision image classification model can be trained even with a small sample.

[0116] Referring to FIG4 , an embodiment of the present invention further provides an image classification method, including:

[0117] Step 41: Obtain the image to be classified;

[0118] Step 42: Use an image classification model to classify the image to be classified, wherein the image classification model includes a feature extraction layer, and the feature extraction layer is trained using a training data set generated by N image sample sets, where N is an integer greater than or equal to 2, and each of the image sample sets includes multiple image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories, and the training data set includes multiple positive sample pairs and multiple negative sample pairs, the two image samples in the positive sample pair belong to the same image sample set, and the two image samples in the negative sample pair belong to different image sample sets.

[0119] Referring to FIG5 , an embodiment of the present invention further provides an image classification device 50, comprising:

[0120] An acquisition module 51 is used to acquire an image to be classified;

[0121] The image classification module 52 is used to classify the image to be classified using an image classification model, wherein N is an integer greater than or equal to 2, and the image classification model includes a feature extraction layer, which is trained using a training data set generated by N image sample sets. Each of the image sample sets includes multiple image samples, and image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories. The training data set includes multiple positive sample pairs and multiple negative sample pairs, and the two image samples in the positive sample pair belong to the same image sample set, and the two image samples in the negative sample pair belong to different image sample sets.

[0122] Referring to FIG6 , an embodiment of the present invention further provides a training device 60 for an image classification model, comprising:

[0123] An acquisition module 61 is used to acquire a feature extraction layer of an image classification model to be trained;

[0124] a generating module 62 configured to generate a training data set based on N image sample sets, where N is an integer greater than or equal to 2, each of the image sample sets including a plurality of image samples, image samples belonging to the same image sample set belonging to the same category, and image samples belonging to different image sample sets belonging to different categories, the training data set including a plurality of positive sample pairs and a plurality of negative sample pairs, the two image samples in the positive sample pairs belonging to the same image sample set, and the two image samples in the negative sample pairs belonging to different image sample sets;

[0125] The first training module 63 is configured to train the feature extraction layer using the training data set to obtain a trained feature extraction layer.

[0126] Optionally, the acquisition module 61 is used to remove several layers behind the pre-trained image classification model to obtain the feature extraction layer.

[0127] Optionally, in different image samples in the same image sample set, main objects belonging to the same category are located in different positions.

[0128] Optionally, the generating module 62 is configured to:

[0129] Generate positive sample pairs, where the number of all positive sample pairs is T is an integer greater than or equal to 0 and less than or equal to N-1, is the number of positive sample pairs corresponding to the T-th image sample set, K T is the number of image samples in the T-th image sample set;

[0130] Generate negative sample pairs, where the number of all negative sample pairs is K V is the number of image samples in the Vth image sample set;

[0131] The training data set is formed according to the positive sample pairs and the negative sample pairs.

[0132] Optionally, forming the training data set according to the positive sample pairs and the negative sample pairs includes:

[0133] Determine the batch_size B and the number of iterations I used for training, where batch_size is the total number of positive and negative sample pairs used in each training;

[0134] The serial numbers of all positive sample pairs are Randomly arrange them and combine them in order to get no less than The sequence number set Data of positive sample pairs + ;

[0135] The serial numbers of all negative sample pairs are Randomly arrange them and combine them in order to get no less than The sequence number set Data of negative sample pairs - ;

[0136] According to the sequence number set Data + and the sequence number set Data - , forming the training data set.

[0137] Optionally, the first training module 63 is used to alternately select from the sequence number set Data + and the sequence number set Data - Positive sample pairs and negative sample pairs are selected to train the feature extraction layer.

[0138] Optionally, the first training module 63 is used to:

[0139] For each sample pair in the training data set, the feature extraction layer is used to extract feature maps F0 and F1 of the two image samples in the sample pair respectively; the sample pair includes a positive sample pair and a negative sample pair;

[0140] Determine the position of the feature block with the highest similarity in the feature maps F0 and F1 as the best matching position;

[0141] Determining an error between the similarity of the best matching position and the true value of the sample pair, determining a loss value based on the error, and adjusting parameters of the feature extraction layer based on the loss value;

[0142] If the accuracy of the feature extraction layer reaches a threshold or the loss value no longer decreases or the number of iterations reaches a preset value, the training is terminated.

[0143] Optionally, determining the position of the feature block with the highest similarity in the feature maps F0 and F1 as the best matching position includes:

[0144] Determine the size and scanning step of the matching feature block, where the side length of the matching feature block is H is the feature height output by the feature extraction layer, W is the feature width output by the feature extraction layer, and R is the ratio of the area of ​​the main object in the image sample to the entire image area;

[0145] According to the matching feature block and the scanning step size, each feature block of size C*BS*BS is traversed in the feature maps F0 and F1 from the two dimensions of feature width and feature height, and the similarity of the feature blocks in the feature maps F0 and F1 is determined, and the position of the feature block with the highest similarity in the feature maps F0 and F1 is selected as the best matching position, where C is the number of channels output by the feature extraction layer.

[0146] Optionally, the image classification model training device 60 further includes:

[0147] The second training module is used to use the N image sample sets to train the layers of the image classification model to be trained except the trained feature extraction layer to obtain the trained image classification model.

[0148] Please refer to Figure 7. An embodiment of the present invention further provides an electronic device 70, including a processor 71, a memory 72, and a computer program stored in the memory 72 and executable on the processor 71. When the computer program is executed by the processor 71, the various processes of the above-mentioned image classification method or the image classification model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0149] An embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the various processes of the above-mentioned image classification method or image classification model training method embodiment, and can achieve the same technical effect. To avoid repetition, it is not described here. The non-transitory computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0150] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0152] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. An image classification method, It is characterized in that include: Get the image to be classified; An image classification model is used to classify the image to be classified, wherein the image classification model includes a feature extraction layer, and the feature extraction layer is trained using a training data set generated by N image sample sets, N is an integer greater than or equal to 2, each of the image sample sets includes multiple image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories, the training data set includes multiple positive sample pairs and multiple negative sample pairs, the two image samples in the positive sample pair belong to the same image sample set, and the two image samples in the negative sample pair belong to different image sample sets.

2. A training method for an image classification model, It is characterized in that include: Get the feature extraction layer of the image classification model to be trained; Generate a training data set according to N image sample sets, wherein N is an integer greater than or equal to 2, each of the image sample sets includes multiple image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories, the training data set includes multiple positive sample pairs and multiple negative sample pairs, two image samples in the positive sample pairs belong to the same image sample set, and two image samples in the negative sample pairs belong to different image sample sets; The feature extraction layer is trained using the training data set to obtain a trained feature extraction layer.

3. The method according to claim 2, It is characterized in that The step of obtaining a feature extraction layer of an image classification model to be trained comprises: The feature extraction layer is obtained by removing several layers behind the pre-trained image classification model.

4. The method according to claim 2, It is characterized in that In different image samples of the same image sample set, main objects belonging to the same category are located in different positions.

5. The method according to claim 2, It is characterized in that Generating a training data set according to N image sample sets includes: Generate positive sample pairs, where the number of all positive sample pairs is T is an integer greater than or equal to 0 and less than or equal to N-1, is the Tth image sample set corresponding to The number of positive sample pairs, K T is the number of image samples in the Tth image sample set; Generate negative sample pairs, where the number of all negative sample pairs is K V is the number of image samples in the Vth image sample set; The training data set is formed according to the positive sample pairs and the negative sample pairs.

6. The method according to claim 5, It is characterized in that The forming the training data set according to the positive sample pairs and the negative sample pairs comprises: Determine the batch_size B and the number of iterations I used for training, where batch_size is the total number of positive sample pairs and negative sample pairs used in each training; The serial numbers of all positive sample pairs are Randomly arrange them and combine them in order to get no less than The sequence number set Data of positive sample pairs + ; Sum the serial numbers of all negative sample pairs Randomly arrange them and combine them in order to get no less than The serial number set Data of negative sample pairs - ; According to the sequence number set Data + and the sequence number set Data - , forming the training data set.

7. The method according to claim 6, It is characterized in that The adopting the training data set to train the feature extraction layer comprises: Alternately from the sequence number set Data + and the sequence number set Data - Positive sample pairs and negative sample pairs are selected to train the feature extraction layer.

8. The method according to claim 2, It is characterized in that The step of training the feature extraction layer using the training data set to obtain a trained feature extraction layer includes: For each sample pair in the training data set, the feature extraction layer is used to extract the feature maps F of the two image samples in the sample pair. 0 and F 1 ; The sample pairs include positive sample pairs and negative sample pairs; Determine the feature map F 0 and F 1 The position of the feature block with the highest similarity is taken as the best matching position; Determine the error between the similarity of the best matching position and the true value of the sample pair, determine a loss value according to the error, and adjust the parameters of the feature extraction layer according to the loss value; If the accuracy of the feature extraction layer reaches a threshold or the loss value no longer decreases or the number of iterations reaches a preset value, the training is terminated.

9. The method according to claim 8, It is characterized in that The determining of the feature map F 0 and F 1 The positions of the feature blocks with the highest similarity as the best matching positions include: Determine the size and scanning step of the matching feature block, where the side length of the matching feature block H is the feature height output by the feature extraction layer, W is the feature width output by the feature extraction layer, and R is the ratio of the area of ​​the main object in the image sample to the entire image area; According to the matching feature block and the scanning step length, in the feature map F 0 and F 1 In the above example, each feature block of size C*BS*BS is traversed from the feature width and feature height dimensions respectively, and the feature map F is determined. 0 and F 1 The similarity of the feature blocks in the feature map F is selected. 0 and F 1 The position of the feature block with the highest similarity in is taken as the best matching position, where C is the number of channels output by the feature extraction layer.

10. The method according to claim 2, It is characterized in that The trained feature extraction layer further includes: The N image sample sets are used to train the layers of the image classification model to be trained except the trained feature extraction layer to obtain the trained image classification model.

11. An image classification device, It is characterized in that include: An acquisition module, used for acquiring images to be classified; An image classification module is used to classify the to-be-classified image using an image classification model, wherein N is an integer greater than or equal to 2, the image classification model includes a feature extraction layer, the feature extraction layer is trained using a training data set generated by N image sample sets, each of the image sample sets includes multiple image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories, the training data set includes multiple positive sample pairs and multiple negative sample pairs, the two image samples in the positive sample pair belong to the same image sample set, and the two image samples in the negative sample pair belong to different image sample sets.

12. A training device for an image classification model, It is characterized in that include: An acquisition module, used to acquire a feature extraction layer of an image classification model to be trained; A generating module, configured to generate a training data set according to N image sample sets, wherein N is an integer greater than or equal to 2, each of the image sample sets includes a plurality of image samples, image samples belonging to the same image sample set belong to the same category, and image samples belonging to different image sample sets belong to different categories, the training data set includes a plurality of positive sample pairs and a plurality of negative sample pairs, the two image samples in the positive sample pairs belong to the same image sample set, and the two image samples in the negative sample pairs belong to different image sample sets; The first training module is used to train the feature extraction layer using the training data set to obtain a trained feature extraction layer.

13. An electronic device, It is characterized in that include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the image classification method as described in claim 1, or the program, when executed by the processor, implements the steps of the image classification model training method as described in any one of claims 2-11.

14. A non-transitory computer-readable storage medium, It is characterized in that The non-volatile computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image classification method as described in claim 1; or, when executed by a processor, implements the steps of the image classification model training method as described in any one of claims 2 to 11.

Images