Image classification model training method, device, electronic device and storage medium

Abstract

The present disclosure provides a training method, apparatus, electronic device, and storage medium for an image classification model, the method comprising: determining at least two target positions on a hypersphere based on the number of image categories included in a first training set; generating a second training set; confirming features corresponding to the first training set and features corresponding to the second training set based on a first encoder; confirming image categories corresponding to at least two target positions based on features corresponding to each image category in the first training set; determining a loss function for the first encoder based on features corresponding to the first training set, features corresponding to the second training set, and image categories corresponding to the at least two target positions; adjusting parameters of the first encoder based on the loss function; and training the second encoder based on a third training set; wherein the third training set includes or excludes the first training set, and the number of image categories in the third training set is greater than the number of image categories in the first image training set.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a training method, device, electronic device, and storage medium for an image classification model. Background Art

[0002] In related technologies, when training image classification models corresponding to images with similar features but different image categories, the image categories of the training data are often unbalanced. A large number of image categories can dominate the training process and change the decision boundaries of a small number of image categories, resulting in an uneven distribution of features of each image category in the final learned feature space. Summary of the Invention

[0003] The present disclosure provides a training method, device, electronic device and storage medium for an image classification model to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a method for training an image classification model is provided, comprising:

[0005] determining at least two target locations on the hypersphere based on the number of image categories included in the first training set;

[0006] Performing image enhancement on the first sample image in the first training set to generate a second training set;

[0007] Determining, based on a first encoder included in the image classification model, features corresponding to each sample image in the first training set and features corresponding to each sample image in the second training set;

[0008] Determining the image categories corresponding to the at least two target positions based on features of sample images corresponding to each image category in the first training set;

[0009] determining a loss function of the first encoder based on features of each sample image included in the first training set, features of each sample image included in the second training set, and image categories corresponding to the at least two target positions;

[0010] adjusting parameters of the first encoder based on the loss function of the first encoder, and confirming that the first encoder with the adjusted parameters is the second encoder;

[0011] Inputting the third training set into the second encoder to train the second encoder, and confirming that the trained second encoder is the third encoder;

[0012] The third training set includes or excludes the first training set, and the number of image categories in the third training set is greater than the number of image categories in the first image training set.

[0013] In the above solution, the method further includes:

[0014] Inputting the fourth training set into the third encoder to train the third encoder, and confirming that the third encoder is the fourth encoder after the training;

[0015] The fourth training set includes the first training set and the third training set, and the number of image categories in the fourth training set is greater than the number of image categories in the third training set.

[0016] In the above solution, the method further includes:

[0017] Inputting the third training set into the third encoder and the linear classification layer included in the image classification model, and determining that the output of the linear classification layer is the predicted image category corresponding to each sample image in the third training set;

[0018] Adjusting the parameters of the linear classification layer based on the image category corresponding to each sample image in the third training set and the predicted image category;

[0019] Alternatively, the fourth training set is input into the fourth encoder and a linear classification layer included in the image classification model, and an output of the linear classification layer is determined to be a predicted image category corresponding to each sample image in the fourth training set;

[0020] Based on the image category corresponding to each sample image in the fourth training set and the predicted image category, the parameters of the linear classification layer are adjusted.

[0021] In the above solution, determining at least two target positions on the hypersphere based on the number of image categories included in the first training set includes:

[0022] At least two target positions on the hypersphere are determined based on the number of image categories included in the first training set and a minimization formula.

[0023] In the above solution, performing image enhancement on the first sample image in the first training set to generate the second training set includes:

[0024] Perform at least one of random cropping, rotation, random blocking, and color change on the first sample image to generate at least two non-identical images as sample images in the second training set.

[0025] In the above solution, the first encoder included in the image classification model confirms the features corresponding to each sample image in the first training set and the features corresponding to each sample image in the second training set, including:

[0026] Inputting each sample image included in the first training set and the image category corresponding to each sample image included in the first training set into a first encoder included in the image classification model, and outputting features of each sample image included in the first training set;

[0027] The sample images included in the second training set and the image categories corresponding to the sample images included in the second training set are input into the first encoder, and the features of the sample images included in the second training set are output.

[0028] In the above solution, determining the image categories corresponding to the at least two target positions based on the features of the sample images corresponding to each image category in the first training set includes:

[0029] Determining the position of the center of each image category on the hypersphere based on features of the sample images corresponding to each image category in the first training set;

[0030] The image categories corresponding to the at least two target positions are determined based on the position of the center of each image category on the hypersphere.

[0031] In the above solution, determining the position of the center of each image category on the hypersphere based on the features of the sample images corresponding to each image category in the first training set includes performing the following operations on the features of the sample images corresponding to each image category:

[0032] Determine an average value of features of the sample images corresponding to the first image category as a correction value;

[0033] determining a predicted position of a center of the first image category on the hypersphere based on a sum of features of sample images corresponding to the first image category and a Euclidean distance between features of the sample images corresponding to the first image category;

[0034] Based on the predicted position and the correction value, the position of the center of the first image class on the hypersphere is determined.

[0035] In the above solution, determining the image categories corresponding to the at least two target positions based on the position of the center of each image category on the hypersphere includes:

[0036] It is determined that the image category of a first target position among the at least two target positions is an image category corresponding to a center on the hypersphere that is shortest in distance from the first target position.

[0037] In the above solution, determining the loss function of the first encoder based on the features of each sample image included in the first training set, the features of each sample image included in the second training set, and the image categories corresponding to the at least two target positions includes:

[0038] Determine the loss function of the first encoder based on the features of the first sample image, the features of each sample image included in the second training set, the features of other sample images in the first training set except the first sample image, the features of at least one second sample image in the first training set that belongs to the same image category as the first sample image, the at least two target positions, and the target position corresponding to the image category of the first sample image.

[0039] In the above solution, determining the loss function of the first encoder based on the features of each sample image included in the first training set, the features of each sample image included in the second training set, and the image categories corresponding to the at least two target positions includes:

[0040] confirming standard contrast loss values based on a feature of the first sample image, features of each sample image included in the second training set, features of at least one second sample image, the number of the at least one second sample image, features of other sample images in the first training set except the first sample image, and at least two target positions;

[0041] Determining a position contrast loss value based on a feature of the first sample image, a target position corresponding to the image category of the first sample image, features of each sample image included in the second training set, features of other sample images in the first training set except the first sample image, and at least two target positions;

[0042] A loss function of the first encoder is determined based on the number of sample images in the first training set, the standard contrast loss value, and the position contrast loss value.

[0043] In the above solution, adjusting the parameters of the first encoder based on the loss function of the first encoder, and confirming that the first encoder with adjusted parameters is the second encoder, includes:

[0044] Adjusting parameters of the first encoder based on a loss function of the encoder;

[0045] The first encoder after parameter adjustment is retrained based on the first training set, and a loss function is determined until the loss function satisfies a first condition, and the training of the first encoder is confirmed to be completed, and the encoder becomes the second encoder.

[0046] In the above solution, inputting the third training set into the second encoder, training the second encoder, and confirming that the trained second encoder is the third encoder includes:

[0047] determining at least three target locations on the hypersphere based on the number of image categories included in the third training set;

[0048] performing image enhancement on the third sample image in the third training set to generate a fifth training set;

[0049] confirming, based on the second encoder included in the image classification model, features corresponding to each sample image in the third training set and features corresponding to each sample image in the fifth training set;

[0050] Determining the image categories corresponding to the at least three target positions based on features of the sample images corresponding to each image category in the third training set;

[0051] determining a loss function of the second encoder based on features of each sample image included in the third training set, features of each sample image included in the fifth training set, and image categories corresponding to the at least three target positions;

[0052] Based on the loss function of the second encoder, the parameters of the second encoder are adjusted, and the second encoder with the adjusted parameters is confirmed to be the third encoder.

[0053] In the above solution, the second image category and the third image category in the fourth training set are the fourth image category in the first training set; the fifth image category, the sixth image category and the seventh image category in the fourth training set are the eighth image category in the first training set;

[0054] In the fourth training set, the ninth image category and the tenth image category are the eleventh image category in the third training set.

[0055] According to a second aspect of the present disclosure, there is provided an image classification method, which uses an image classification model trained based on the first aspect, and the method includes:

[0056] Inputting the image to be classified into an encoder included in the image classification model, and confirming that the output of the encoder is a feature of the image to be classified; the feature of the image to be classified is reflected in a hypersphere;

[0057] Inputting the features of the image to be classified into a linear classification layer included in the image classification model, and confirming that the output of the linear classification layer is the image category of the image to be classified;

[0058] Wherein, the encoder includes a third encoder or a fourth encoder.

[0059] According to a third aspect of the present disclosure, there is provided a training device for an image classification model, comprising:

[0060] a target position determining unit, configured to determine at least two target positions on the hypersphere based on the number of image categories included in the first training set;

[0061] a training set enhancement unit, configured to perform image enhancement on the first sample image in the first training set to generate a second training set;

[0062] a first confirmation unit, configured to confirm, based on a first encoder included in the image classification model, features corresponding to each sample image in the first training set and features corresponding to each sample image in the second training set;

[0063] a second confirmation unit, configured to confirm the image categories corresponding to the at least two target positions based on features of sample images corresponding to each image category in the first training set;

[0064] a loss determining unit, configured to determine a loss function of the first encoder based on features of each sample image included in the first training set, features of each sample image included in the second training set, and image categories corresponding to the at least two target positions;

[0065] an adjusting unit, configured to adjust parameters of the first encoder based on a loss function of the first encoder, and confirm that the first encoder whose parameters are adjusted is a second encoder;

[0066] an iterative unit, configured to input the third training set into the second encoder, train the second encoder, and confirm that the trained second encoder is a third encoder;

[0067] The third training set includes or excludes the first training set, and the number of image categories in the third training set is greater than the number of image categories in the first image training set.

[0068] According to a fourth aspect of the present disclosure, there is provided an image classification apparatus, comprising:

[0069] A first input unit is configured to input an image to be classified into an encoder included in the image classification model, and confirm that an output of the encoder is a feature of the image to be classified, wherein the feature of the image to be classified is reflected in a hypersphere;

[0070] A second input unit is configured to input the features of the image to be classified into a linear classification layer included in the image classification model, and confirm that the output of the linear classification layer is the image category of the image to be classified;

[0071] Wherein, the encoder includes a third encoder or a fourth encoder.

[0072] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0073] at least one processor; and

[0074] a memory communicatively connected to the at least one processor; wherein,

[0075] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0076] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the present disclosure.

[0077] The training method of the image classification model disclosed in the present invention determines at least two target positions on a hypersphere based on the number of image categories included in a first training set; performs image enhancement on a first sample image in the first training set to generate a second training set; confirms features corresponding to each sample image in the first training set and features corresponding to each sample image in the second training set based on a first encoder included in the image classification model; confirms image categories corresponding to the at least two target positions based on features of sample images corresponding to each image category in the first training set; determines a loss function of the first encoder based on features of each sample image included in the first training set, features of each sample image included in the second training set, and image categories corresponding to the at least two target positions; adjusts parameters of the first encoder based on the loss function of the first encoder, and confirms that the first encoder with adjusted parameters is a second encoder. ; Input the third training set into the second encoder, train the second encoder, and confirm that the trained second encoder is the third encoder; wherein, the third training set includes or does not include the first training set, and the number of image categories in the third training set is greater than the number of image categories in the first image training set; on the one hand, introduce the concept of hypersphere, and after determining at least two target positions on the hypersphere, further confirm the image types of the at least two target positions, so that the centers of different image types are evenly distributed on a hypersphere, thereby obtaining a clear decision boundary between image types, avoiding errors caused by imbalanced image categories of sample images in subsequent training, and improving the accuracy of the encoder; on the other hand, during training, first train the first training set with fewer image categories, and then train the second training set with more image categories, and gradually enhance the expression ability of the model through training from easy to difficult.

[0078] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0080] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

[0081] Figure 1 An optional flow chart of a method for training an image classification model provided by an embodiment of the present disclosure is shown;

[0082] Figure 2 Another optional flow chart of the training method of the image classification model provided by the embodiment of the present disclosure is shown;

[0083] Figure 3 A schematic diagram of a hypersphere provided by an embodiment of the present disclosure is shown;

[0084] Figure 4 Another optional flow chart of the training method of the image classification model provided by the embodiment of the present disclosure is shown;

[0085] Figure 5 An optional flow chart of an image classification method provided by an embodiment of the present disclosure is shown;

[0086] Figure 6 An optional structural diagram of a training device for an image classification model provided by an embodiment of the present disclosure is shown;

[0087] Figure 7 An optional structural diagram of an image classification device provided by an embodiment of the present disclosure is shown;

[0088] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0089] To make the purposes, features, and advantages of the present disclosure more apparent and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative work shall fall within the scope of protection of the present disclosure.

[0090] Figure 1 An optional flow chart of the training method of the image classification model provided by the embodiment of the present disclosure is shown, and will be explained according to each step.

[0091] Step S101 : determining at least two target positions on a hypersphere based on the number of image categories included in a first training set.

[0092] In some embodiments, the training set used for training may be a fourth training set, the fourth training set including the first training set and the third training set, and the number of sample images in the first training set may be equal to the number of sample images in the second training set. The second image category (a1) and the third image category (a2) in the fourth training set are the fourth image category (a'1) in the first training set; the fifth image category (a3), the sixth image category (a4), and the seventh image category (a5) in the fourth training set are the eighth image category (a'2) in the first training set; and the ninth image category (a2) and the tenth image category (a3) in the fourth training set are the eleventh image category (a'3) in the third training set.

[0093] Specifically, the image categories of the sample images in the fourth training set can be divided into a1, a2, ..., a n , a total of n (n is an integer greater than or equal to 3) image categories; for the first training set, at least two image categories with similar features can be merged to form a new image category. For example, when n=6, a1 and a2 are merged into a'1, and a3, a4, and a5 are merged into a'2. Then the number of image categories in the first training set is 3, namely a'1, a'2, and a6. For the third training set, at least two image categories with similar features can also be merged, but the number of image categories finally formed is greater than the number of image categories in the first training set and less than the number of image categories in the fourth training set. Still taking n=6 as an example, a2 and a3 are merged into a'3, and a5 and a6 are merged into a'4. Then the number of image categories in the third training set is 4, namely a1, a'3, a4, and a'4. Optionally, the image categories to be merged are determined according to needs or experimental results. For the first training set and the third training set, the image categories may be merged in different ways or completely different ways.

[0094] In the present disclosure, the image classification model is first trained based on the first training set, and then iteratively trained based on the third training set and the fourth training set. The expression ability of the image classification model can be improved by training from easy to difficult.

[0095] In some embodiments, a training device for an image classification model (hereinafter referred to as the device) determines at least two target locations on a hypersphere based on the number of image categories included in the first training set and a minimization formula, wherein the graph formed by the at least two target locations is a positive simplex corresponding to the hypersphere, and the at least two target locations are vertices of the positive simplex and are located on the hypersphere corresponding to the hypersphere. Optionally, the device can also determine the at least two target locations based on the dimension corresponding to the hypersphere (the dimension of the feature space), and the dimension corresponding to the hypersphere can be determined based on actual needs or experimental results.

[0096] Each target position corresponds to an image category, and the number of target positions is equal to the number of image categories included in the first training set.

[0097] Step S102: performing image enhancement on the first sample image in the first training set to generate a second training set.

[0098] In some embodiments, the device performs image enhancement on the first sample image in the first training set to generate a second training set; wherein, the first sample image is any sample image in the first training set, and the device can perform image enhancement on all sample images in the first training set, and generate the enhanced sample images into different training sets based on the sample images.

[0099] In specific implementation, the device can perform at least one of random cropping, rotation, random occlusion and color change on the first sample image to generate at least two non-identical images as sample images in the second training set; each sample image in the first training set can correspond to a second training set separately, that is, the total number of the second training sets is equal to the number of sample images in the first training set, and they together constitute the first enhanced training set.

[0100] Among them, the first sample image and any sample image in the corresponding second training set (that is, the training set generated based on the enhancement of the first sample image) are mutually positive examples, and are mutually negative examples with sample images other than the first sample image in the first training set, and are mutually negative examples with any sample image in the second training set generated by enhancing other sample images (sample images other than the first sample image in the first training set).

[0101] Step S103 : confirming features corresponding to each sample image in the first training set and features corresponding to each sample image in the second training set based on the first encoder included in the image classification model.

[0102] In some embodiments, the device inputs each sample image included in the first training set and the image category corresponding to each sample image included in the first training set into the first encoder included in the image classification model, and outputs the features of each sample image included in the first training set; inputs each sample image included in the second training set and the image category corresponding to each sample image included in the second training set into the first encoder, and outputs the features of each sample image included in the second training set.

[0103] The features of each sample image included in the first training set or the features of each sample image included in the second training set are features in the feature space, which can be reflected in the hypersphere, and specifically, can be distributed on the hypersphere surface corresponding to the hypersphere.

[0104] Step S104 : confirming the image categories corresponding to the at least two target positions based on features of the sample images corresponding to each image category in the first training set.

[0105] In some embodiments, the device confirms the position of the center of each image category on the hypersphere based on the features of the sample images corresponding to each image category in the first training set; and confirms the image categories corresponding to the at least two target positions respectively (i.e., one target position corresponds to one image category) based on the position of the center of each image category on the hypersphere.

[0106] During specific implementation, the device confirms the average value of the features of the sample images corresponding to the first image category as a correction value; based on the sum of the features of the sample images corresponding to the first image category and the Euclidean distance between the features of the sample images corresponding to the first image category, confirms the predicted position of the center of the first image category on the hypersphere; based on the predicted position and the correction value, confirms the position of the center of the first image category on the hypersphere.

[0107] Specifically, the device can determine the sum of the features of the sample images corresponding to the first image category and the quotient of the Euclidean distances between the features of the sample images corresponding to the first image category as the predicted position, correct the predicted position based on the correction value, and confirm that the corrected position is the position of the center of the first image category on the hypersphere.

[0108] In a specific implementation, the apparatus may determine that the image category of a first target location among the at least two target locations is the image category corresponding to the center of the hypersphere that is shortest in distance from the first target location. Specifically, the apparatus may assign an image category to each target location based on a Hungarian matching algorithm by minimizing the distance between the target location and the center of the image category assigned to the target location.

[0109] For example, if the first target position is closest to the center of the first image category, then the image category of the first target position is the first image category. The distance here can be Euclidean distance or distance on a hypersphere.

[0110] Step S105 : determining a loss function of the first encoder based on features of each sample image included in the first training set, features of each sample image included in the second training set, and image categories corresponding to the at least two target positions.

[0111] In specific implementation, the device can determine the loss function of the first encoder based on the features of the first sample image, the features of each sample image included in the second training set, the features of other sample images other than the first sample image in the first training set, the features of at least one second sample image in the first training set that belongs to the same image category as the first sample image, the at least two target positions, and the target position corresponding to the image category of the first sample image.

[0112] Specifically, the device confirms a standard contrast loss value based on the features of the first sample image, the features of each sample image included in the second training set, the features of at least one second sample image, the number of the at least one second sample image, the features of other sample images in the first training set except the first sample image, and at least two target positions; confirms a position contrast loss value based on the features of the first sample image, the target position corresponding to the image category of the first sample image, the features of each sample image included in the second training set, the features of other sample images in the first training set except the first sample image, and at least two target positions; confirms the loss function of the first encoder based on the number of sample images in the first training set, the standard contrast loss value, and the position contrast loss value.

[0113] Further, the device can determine a first exponent of the product of the transpose of the features of the first sample image and the features of any sample image in the set consisting of the features of each sample image included in the second training set and the features of at least one second sample image, and a second exponent of the product of the transpose of the features of the first sample image and the features of other sample images other than the first sample image in the first training set, the features of each sample image included in the second training set, and any sample image in the set consisting of the at least two target positions; and confirm the standard contrast loss value based on the logarithm of the quotient of the first index and the second index.

[0114] In addition, the device can determine a third exponent of the product of the transpose of the features of the first sample image and the target position corresponding to the image category of the first sample image, and a second exponent of the product of the transpose of the features of the first sample image and the features of other sample images other than the first sample image in the first training set, the features of each sample image included in the second training set, and any sample image in the set consisting of the at least two target positions; and confirm the position contrast loss value based on the logarithm of the quotient of the third index and the second index.

[0115] Step S106: Adjust parameters of the first encoder based on the loss function of the first encoder, and confirm that the first encoder with adjusted parameters is the second encoder.

[0116] In some embodiments, the apparatus adjusts the parameters of the first encoder based on the loss function of the first encoder, and then repeats steps S101 to S106, or steps S103 to S106, with each repetition confirming the loss function based on the features output by the encoder with the current parameters, until the loss function satisfies a first condition, then confirming that the training of the first encoder is complete and the encoder is the second encoder; wherein the first condition can be set based on actual needs or experimental results, such as when the loss value corresponding to the loss function is less than a first threshold. It should be understood by those skilled in the art that this is only an example and is not intended to limit the scope of the present disclosure.

[0117] Step S107: input the third training set into the second encoder, train the second encoder, and confirm that the trained second encoder is the third encoder.

[0118] According to step S101 , the number of image categories in the third training set is greater than the number of image categories in the first training set, so the positive simplex corresponding to the hypersphere and the vertices corresponding to the positive simplex need to be re-determined.

[0119] In a specific implementation, the apparatus determines at least three target positions of a positive simplex corresponding to the third training set based on the number of image categories in the third training set and the dimension of the hypersphere. Then, steps S102 to S106 are repeated, inputting the third training set into the second encoder to train the second encoder. Based on the features corresponding to each sample image included in the third training set and the at least three target positions, a loss function of the second encoder is determined. In response to the loss function corresponding to the second encoder satisfying a second condition, the training of the second encoder is determined to be complete, and the second encoder is designated as the third encoder.

[0120] In some optional embodiments, if the third training set includes the first training set (that is, the image categories in the third training set are first divided into b as the first training set for training, and then the third training set is trained, where b is less than the number of image categories included in the third training set), then it is determined that the training of the third encoder is completed; if the third training set does not include the first training set, training is performed based on the fourth training set. After the training of the fourth training set is completed, it is confirmed that the encoder training is completed.

[0121] Step S108: training the linear classification layer included in the image classification model.

[0122] In some embodiments, if the third training set includes the first training set, the third training set is input into the third encoder and the linear classification layer included in the image classification model, and the output of the linear classification layer is determined to be the predicted image category corresponding to each sample image in the third training set; based on the image category corresponding to each sample image in the third training set and the predicted image category, the parameters of the linear classification layer are adjusted.

[0123] In other embodiments, if the third training set does not include the first training set, the fourth training set is input into the fourth encoder and the linear classification layer included in the image classification model, and the output of the linear classification layer is determined to be the predicted image category corresponding to each sample image in the fourth training set; based on the image category corresponding to each sample image in the fourth training set and the predicted image category, the parameters of the linear classification layer are adjusted.

[0124] In this way, the training method of the image classification model provided by the embodiment of the present disclosure, on the one hand, introduces the concept of a hypersphere, and after determining at least two target positions on the hypersphere, further confirms the image types of the at least two target positions, so that the centers of different image types are evenly distributed on a hypersphere, thereby obtaining a clear decision boundary between image types, avoiding errors caused by unbalanced image categories of sample images in subsequent training processes, and improving the accuracy of the encoder; secondly, during training, the first training set with fewer image categories is first trained, and then the second training set with more image categories is trained, and the expression ability of the model is gradually enhanced through training from easy to difficult.

[0125] Figure 2 Another optional flow chart of the training method of the image classification model provided by the embodiment of the present disclosure is shown, and will be explained according to each step.

[0126] In the disclosed embodiment, the image classification model is trained in an iterative manner, and the image categories in the training set used are trained from few to many. During each iteration, the sample images in the training set used may be the same or different; but the training set used in the last iteration is the sum of all the previous training sets. For example, taking two iterations as an example, the training sets used in the two iterations are the same, but the sample images in the training set are divided into c categories (c image categories) in the first iteration, and the sample images in the training set are divided into f categories (f image categories) in the second iteration, where c < f; taking three iterations as an example, the training sets used in the first and second iterations are different, and the training set used in the third iteration is the sum of the training sets used in the first two iterations, and the image category division of the sample images is gradually increased; taking four iterations as an example, the training sets used in the first three iterations are different, and the training set used in the fourth iteration is the sum of the training sets used in the first three iterations, and the image category division of the sample images is gradually increased.

[0127] In the embodiment of the present disclosure, three iterations are taken as an example for description. The division of the training set and the division of the image categories in the training set can refer to step S101.

[0128] Step S201: training a first encoder included in an image classification model based on a first training set to obtain a trained second encoder.

[0129] In some embodiments, step S201 is the same as steps S101 to S106 and will not be repeated here.

[0130] Step S202: Train the second encoder based on the third training set to obtain a trained third encoder.

[0131] In some embodiments, similar to step S201, taking the classification of images in the third training set into four image categories as an example, the apparatus determines the hypersphere S based on the number of image categories in the third training set and the minimization formula. d-1 ={u∈R d : ‖u‖=1} on the four target positions C = 4, d is the dimension of the hypersphere, which can be achieved by the following formula:

[0132]

[0133] Where τ is a temperature parameter that can be set based on actual needs or experimental results. L1 is the uniformity loss. The distribution optimization of positive simplex vertices on the hypersphere should converge to a uniform distribution. u is a unit vector in d-dimensional space with a vector length of 1.

[0134] Figure 3 A schematic diagram of a hypersphere provided by an embodiment of the present disclosure is shown.

[0135] like Figure 3 As shown, a positive simplex is a figure where the distances from the center of the hypersphere to the vertex of the upper positive simplex are equal. If the training set includes two image categories, the target positions to be determined are two, such as Figure 3 As shown in (1), ζ1 and ζ2 are included in the training set; if the training set includes 3 image categories, the number of target locations to be determined is 3, as shown in Figure 3 As shown in (2), ζ1, ζ2 and ζ3 are included in the training set; if the training set includes 4 image categories, the number of target locations to be determined is 4, as shown in Figure 3 As shown in (3), ζ1, ζ2, ζ3 and ζ4 are included.

[0136] In some embodiments, the apparatus performs sample enhancement on the third sample images in the third training set, and determines a set of enhanced sample images as a fifth training set.

[0137] Specifically, each sample image in the third training set can correspond to a fifth training set individually, that is, the total number of the fifth training set is equal to the number of sample images in the third training set, and they together constitute the second enhanced training set; the third sample image and any sample image in the corresponding fifth training set (the training set generated based on the enhancement of the third sample image) are each other's positive examples, and are each other's negative examples with the sample images in the third training set other than the third sample image, and are each other's negative examples with any sample image in the fifth training set generated by enhancing other sample images (sample images in the third training set other than the third sample image).

[0138] In some embodiments, the device assigns an image category to each target location (the four target locations mentioned above), specifically:

[0139] The device inputs the third training set and the fifth training set into the second encoder, and outputs features corresponding to each sample image in the third training set and features corresponding to each sample image in the fifth training set.

[0140] Then, the feature center c′ of each image category is determined based on the features corresponding to each sample image in the third training set. k (correction value), k∈[1, C]. That is, the correction value is obtained by averaging the features of multiple sample images belonging to the same image category; the predicted position c of the center of the first image category on the hypersphere is determined based on the sum of the features of the sample images corresponding to the second image category and the Euclidean distance between the features of the sample images corresponding to the second image category. k , which can be determined based on the following formula:

[0141]

[0142] Among them, v j is the feature of the sample image corresponding to any second image category, F k It is the feature set of category k in the third training set (batch) after the second encoder extracts features.

[0143] Based on the weighted moving average method, the predicted position c is updated k , which can be updated based on the following methods:

[0144] c k =(1-α)*c k +α*c′ k (3)

[0145] Wherein, α is a weight coefficient, which can be set based on actual needs or experimental results, such as α = 0.1. It should be understood by those skilled in the art that this is only an example and is not intended to limit the present disclosure.

[0146] In some embodiments, the apparatus assigns a class to each target location by minimizing the distance between the target location and the center of the class assigned to them according to the Hungarian matching algorithm.

[0147]

[0148] In a specific implementation, the device may determine the image category of the second target location among the four target locations as the image category corresponding to the center of the hypersphere that is shortest from the second target location. For example, if the second target location is closest to the center of the second image category, the image category of the second target location is the second image category. The distance here can be Euclidean distance or distance on the hypersphere.

[0149] Furthermore, the device may determine the loss function of the second encoder based on features of each sample image included in the third training set, features of each sample image included in the fifth training set, and image categories corresponding to the four target positions.

[0150] In a specific implementation, a standard contrast loss value is confirmed based on the features of the third sample image, the features of each sample image included in the fourth training set, the features of at least one fourth sample image (the sample image in the fifth training set), the number of the at least one fourth sample image, the features of the sample images other than the third sample image in the third training set, and the four target positions; a position contrast loss value is confirmed based on the features of the third sample image, the target position corresponding to the image category of the third sample image, the features of each sample image included in the fifth training set, the features of the sample images other than the third sample image in the third training set, and the four target positions; and a loss function of the second encoder is confirmed based on the number of sample images in the third training set, the standard contrast loss value, and the position contrast loss value:

[0151]

[0152] Among them, the sample images in the third training set y i Representative sample image (third sample image) x i image category, N is the total number of sample images in the third training set; v i is the third sample image x i In the unit hypersphere S d-1 The above features, is the third sample image x i The corresponding fifth training set (expanded set) The characteristics of the data in V i is the number of the third training set (current batch) except v i Other features, is with x i m data features belonging to the same image category. And is the corresponding image category y i The target position; further, is any element in the set consisting of the features corresponding to each sample image in the fifth training set and the features corresponding to sample images of the same image category as the third sample image in the third training set; j It is any element in a set consisting of features corresponding to each sample image in the fifth training set, features corresponding to other sample images except the third sample image in the third training set, and the four target positions.

[0153] The first part of formula (5) is the unstandardized contrast loss, where the numerator encourages the same image category to be as close as possible in the feature space, and the denominator encourages different image categories to be as far away as possible in the feature space; the second part is the position contrast loss value (the contrast loss between the sample and the target position), which makes the features of the image sample closer to the target position corresponding to its image category and away from the target positions corresponding to other image categories.

[0154] L2 ensures that the projection of each image category in the feature space is aligned with the assigned target position, and at the same time evenly distributes the targets on the hypersphere, which helps to improve the problem of poor training effect caused by the imbalanced distribution of image categories in the training set.

[0155] Then, step S202 is repeated to train the second encoder. Each repetition confirms a loss function based on the features output by the encoder with the current parameters until the loss function satisfies a second condition. The second encoder training is then confirmed to be complete and becomes the third encoder. The second condition can be set based on actual needs or experimental results, such as when the loss value corresponding to the loss function is less than a second threshold. It should be understood by those skilled in the art that this is merely an example and is not intended to limit the scope of this disclosure.

[0156] Step S203: Train the third encoder based on the fourth training set to obtain a trained fourth encoder.

[0157] The specific steps of step S203 are similar to those of step S201 or step S202, except that the training set is changed, and the number of image categories in the fourth training set is greater. The target position on the hypersphere needs to be reconfirmed based on step S201 or step S202, which will not be repeated here.

[0158] Step S204: training the linear classification layer included in the image classification model.

[0159] In some embodiments, if the third training set includes the first training set, the third training set is input into the third encoder and the linear classification layer included in the image classification model, and the output of the linear classification layer is determined to be the predicted image category corresponding to each sample image in the third training set; based on the image category corresponding to each sample image in the third training set and the predicted image category, the parameters of the linear classification layer are adjusted.

[0160] In other embodiments, if the third training set does not include the first training set, the fourth training set is input into the fourth encoder and the linear classification layer included in the image classification model, and the output of the linear classification layer is determined to be the predicted image category corresponding to each sample image in the fourth training set; based on the image category corresponding to each sample image in the fourth training set and the predicted image category, the parameters of the linear classification layer are adjusted.

[0161] In this way, the training method of the image classification model provided by the embodiment of the present disclosure, on the one hand, introduces the concept of a hypersphere, and after determining at least two target positions on the hypersphere, further confirms the image types of the at least two target positions, so that the centers of different image types are evenly distributed on a hypersphere, thereby obtaining a clear decision boundary between image types, avoiding errors caused by unbalanced image categories of sample images in subsequent training processes, and improving the accuracy of the encoder; secondly, during training, the first training set with fewer image categories is first trained, and then the second training set with more image categories is trained, and the expression ability of the model is gradually enhanced through training from easy to difficult.

[0162] In clinical practice, the BIRADS (Beneficial Assessment of Rational Pathology) (BIRADS) classification system is generally used to categorize breast lesions into six categories: {2, 3, 4A, 4B, 4C, and 5}. A BIRADS 2 lesion is typically a cystic, solid, regularly shaped, and well-defined benign lesion with no risk of progression. A BIRADS 3 lesion is a typical clinical benign change with a 2% or less risk of progression. BIRADS 4A, 4B, and 4C lesions typically present with one, two, three, or four malignant features on ultrasound, with a 3% to 10% risk, a 10% to 50% risk, and a 50% to 95% risk of progression, respectively. A BIRADS 5 lesion has a 95% probability of being malignant.

[0163] In medical applications of computer vision, the BIRADS classification of breast lesions begins by acquiring a large amount of breast ultrasound image or video data. Experts label the lesions in the data with BIRADS grades, which serves as the gold standard. A CNN model is then used to extract features and classify the labeled data. After continuous optimization and iteration, the model is fixed. Finally, BIRADS is predicted for any unlabeled data.

[0164] However, in practice, the training data obtained is often class-imbalanced. This is because in the real world, malignant cases obtained through routine screening account for a small fraction, while the majority are benign lesions. Specifically, there is a higher percentage of BIRADS 2 and 3 cases, while 4C and 5 cases are relatively rare.

[0165] In network model training, a large number of categories can dominate the training process and change the decision boundaries of a small number of categories, which leads to an uneven distribution of features for each class in the final learned feature space.

[0166] Secondly, based on the above background knowledge, the distinction boundaries between categories 3 and 4A or categories 4C and 5 are not very clear in clinical practice, so the CNN model is usually not very ideal for classifying these categories.

[0167] Figure 4Another optional flow chart of the training method of the image classification model provided by the embodiment of the present disclosure is shown. The training method of the image classification model provided by the embodiment of the present disclosure will be further explained by taking the images in the training set as breast ultrasound images as an example. Based on the training method provided by the present disclosure, a method is proposed based on a given spatial target position, which uses a contrast supervised learning method to make the center of each class evenly distributed in a hypersphere through iterative optimization, thereby obtaining a clear decision boundary between classes. This solves the problem of low accuracy caused by imbalanced categories in breast ultrasound image training data; the BIRADS six-category task of breast lesions is split, and the model expression ability is gradually enhanced through training from easy to difficult.

[0168] Before model training, it is necessary to calculate the optimal position of different image categories in the feature space based on the number of image categories in the training set, that is, to determine the target position of different image categories on the hypersphere. Assume that the image categories of the sample images in the first training set, the third training set, and the fourth training set are uniform on BIRADS, and the optimal case for convergence of comparative supervised learning is that the feature centers of all categories are located on the unit hypersphere S. d-1 = {u∈R d :‖u‖=1} is the vertex of a positive simplex centered at the center of the sphere, such as Figure 3 When the number of categories C and the dimension d of the feature space are known, the ideal target position can be calculated by minimizing the formula (1):

[0169] Step S401: confirm the training set.

[0170] In some embodiments, using breast ultrasound images as an example, the device splits the BIRADS classification task into three levels, from easy to difficult. The training set (the fourth training set) is also split randomly and evenly into two parts (the first training set and the third training set), with an equal proportion of image categories in each training set.

[0171] The data of categories 2 and 3 in BIRADS are merged into one category, and the data of categories 4A, 4B, and 4C are merged into one category. The six-category task is simplified to the three-category task 2, 3, 4A, 4B, 4C, and 5, and the first training set is used to train this task.

[0172] The data of BIRADS categories 3 and 4A are merged into one category, and the data of categories 4C and 5 are merged into one category. The six-category task is simplified to the four-category task 2, 3, 4A, 4B, 4C, 5, and the third training set is used for training this task.

[0173] All training data are used to perform BIRADS six-category 2 / 3 / 4A / 4B / 4C / 5, and the fourth training set is used to train this task.

[0174] Step S402: training a first encoder based on a first training set.

[0175] In some embodiments, C=3, d=215, the first encoder is resnet50, and three target positions (i.e., the at least two target positions mentioned above) uniformly distributed in the feature space are determined based on formula (1).

[0176]

[0177] Assume that the first training set includes 32 sample images. Each sample image in the first training set is sample enhanced. After the first sample image is enhanced, 10 corresponding second sample images are obtained. The resulting set is the second training set. Specifically, the sample augmentation method may include at least one of random cropping, rotation, random occlusion, and color change. The first sample image is determined to be a positive example with any sample image in the second training set, and a negative example with any sample image in the first training set other than the first sample image.

[0178] In some embodiments, the device is Each target position in is assigned an image category. Specifically:

[0179] First, each sample image in the first training set and the image category corresponding to each sample image, and each sample image in the second training set and the image category corresponding to each sample image are input into the first encoder, and the features of each sample image in the first training set and the features of each sample image in the second training set are output.

[0180] Calculate the average value based on the features of the sample images corresponding to the first image category in the first training set, and confirm that the average value is the first correction value c′ corresponding to the first image category k ; Determine the predicted position c of the center of the first image category on the hypersphere based on the features of the sample image corresponding to the first image category and formula (2) k , based on the first correction value, update the predicted position c of the center of the first image category on the hypersphere k =0.9*c k +0.1*c′ k .

[0181] In some embodiments, the apparatus assigns a class to each target location by minimizing the distance between the target location and the center of the class assigned to them according to the Hungarian matching algorithm, i.e., formula (2).

[0182] In a specific implementation, the device may determine that the image category of the first target location among the three target locations is the image category corresponding to the center of the hypersphere that is shortest from the first target location. For example, if the first target location is closest to the center of the first image category, then the image category of the first target location is the first image category. The distance here can be Euclidean distance or distance on the hypersphere.

[0183] Furthermore, the device optimizes the first encoder based on a weighted combination of standard contrast loss and position contrast loss (the contrast loss between the target position and the sample), that is, formula (5) as the final loss.

[0184] Specifically, each time, sample enhancement is performed on each sample image in the first training set, and the first training set and the multiple training sets obtained after sample enhancement are input into the encoder obtained in the previous round of training, and the features (spatial features) of the sample images included in the first training set and the multiple training sets are extracted. After each extraction, the image categories are reassigned to the three target positions, and the loss function is calculated based on formula (5). The parameters of the encoder obtained in the previous round of training are adjusted (optimized) until the loss function tends to be stable (meets the first condition), and it is confirmed that the training of the first encoder is completed and it is the second encoder.

[0185] Step S403: train the second encoder based on the third training set.

[0186] In some embodiments, the apparatus uses the second encoder as the initial encoder for this iteration, defines C=4, d=512, and repeats the process of step S402 or step S202 to obtain a third encoder with further improved expressiveness.

[0187] Step S404: train the third encoder based on the fourth training set.

[0188] In some embodiments, the apparatus uses the third encoder as the initial encoder for this iteration, defines C=6, d=512, and repeats the process of step S402, step S404 or step S202 to obtain a fourth encoder with further improved expressiveness.

[0189] Through step S402 to step S403, the encoder is trained in a step-by-step process from simple to difficult, so that the image classification model can more easily learn data features of more complex tasks.

[0190] Step S405: training the linear classification layer included in the image classification model.

[0191] In some embodiments, it is confirmed that the parameters of the fourth encoder are fixed, the fourth training set is input into the fourth encoder and the linear classification layer included in the image classification model, and the output of the linear classification layer is determined to be the predicted image category corresponding to each sample image in the fourth training set; a cross-entropy loss function is determined based on the image category corresponding to each sample image in the fourth training set and the predicted image category, and the parameters of the linear classification layer are adjusted based on the cross-entropy loss function.

[0192] In this way, through the training method of the image classification model provided by the embodiment of the present disclosure, the target position of each image category in the feature space is predetermined based on the number of image categories and the dimension of the feature space, and the target position is evenly distributed to each image category during the training process, thereby avoiding the problems caused by imbalanced training data, making the feature space more uniform and the prediction more accurate; in addition, by splitting the breast lesion BIRADS classification task into three tasks from easy to difficult and gradually training them, the expressive ability of the image classification model is improved, making the prediction more accurate.

[0193] Figure 5 An optional flow chart of the image classification method provided by an embodiment of the present disclosure is shown, and will be explained according to each step.

[0194] Step S501: input the image to be classified into an encoder included in the image classification model, and confirm that the output of the encoder is the feature of the image to be classified.

[0195] In some embodiments, the image classification device inputs the image to be classified into the encoder included in the image classification model trained by the above steps S101 to S108, steps S201 to S204 or steps S401 to S405, and confirms that the output of the encoder is the feature of the image to be classified.

[0196] The features of the image to be classified are spatial features, which can be reflected in a hypersphere; and the encoder can be a third encoder or a fourth encoder.

[0197] Step S502: Input the features of the image to be classified into the linear classification layer included in the image classification model, and confirm that the output of the linear classification layer is the image category of the image to be classified.

[0198] Taking the image to be classified as a breast ultrasound image as an example, the output of the linear classification layer is the BIRADS grade corresponding to the breast ultrasound image.

[0199] Figure 6 An optional structural diagram of a training device for an image classification model provided by an embodiment of the present disclosure is shown, and will be explained according to each part.

[0200] In some embodiments, the training device 600 of the image classification model includes a target position determination unit 601, a training set enhancement unit 602, a first confirmation unit 603, a second confirmation unit 604, a loss determination unit 605, an adjustment unit 606 and an iteration unit 607.

[0201] The target position determination unit 601 is configured to determine at least two target positions on the hypersphere based on the number of image categories included in the first training set;

[0202] The training set enhancement unit 602 is configured to perform image enhancement on the first sample image in the first training set to generate a second training set;

[0203] The first confirmation unit 603 is configured to confirm features corresponding to each sample image in the first training set and features corresponding to each sample image in the second training set based on a first encoder included in the image classification model;

[0204] The second confirmation unit 604 is configured to confirm the image categories corresponding to the at least two target positions based on features of the sample images corresponding to each image category in the first training set;

[0205] The loss determining unit 605 is configured to determine a loss function of the first encoder based on features of each sample image included in the first training set, features of each sample image included in the second training set, and image categories corresponding to the at least two target positions;

[0206] The adjusting unit 606 is configured to adjust parameters of the first encoder based on the loss function of the first encoder, and confirm that the first encoder whose parameters are adjusted is the second encoder;

[0207] The iterative unit 607 is configured to input the third training set into the second encoder, train the second encoder, and confirm that the trained second encoder is a third encoder;

[0208] The third training set includes or excludes the first training set, and the number of image categories in the third training set is greater than the number of image categories in the first image training set.

[0209] The iteration unit 607 is further configured to input the fourth training set into the third encoder to train the third encoder, and confirm that the third encoder is a fourth encoder after the training;

[0210] The fourth training set includes the first training set and the third training set, and the number of image categories in the fourth training set is greater than the number of image categories in the third training set.

[0211] The first confirmation unit 603 is further configured to input the third training set into the third encoder and the linear classification layer included in the image classification model, and determine that the output of the linear classification layer is the predicted image category corresponding to each sample image in the third training set; or input the fourth training set into the fourth encoder and the linear classification layer included in the image classification model, and determine that the output of the linear classification layer is the predicted image category corresponding to each sample image in the fourth training set;

[0212] The adjustment unit 606 is further used to adjust the parameters of the linear classification layer based on the image category corresponding to each sample image in the third training set and the predicted image category; or to adjust the parameters of the linear classification layer based on the image category corresponding to each sample image in the fourth training set and the predicted image category.

[0213] The target position determination unit 601 is specifically configured to determine at least two target positions on the hypersphere based on the number of image categories included in the first training set and a minimization formula.

[0214] The training set enhancement unit 602 is specifically configured to perform at least one of random cropping, rotation, random occlusion, and color change on the first sample image to generate at least two non-identical images as sample images in the second training set.

[0215] The first confirmation unit 603 is specifically used to input each sample image included in the first training set and the image category corresponding to each sample image included in the first training set into the first encoder included in the image classification model, and output the features of each sample image included in the first training set; input each sample image included in the second training set and the image category corresponding to each sample image included in the second training set into the first encoder, and output the features of each sample image included in the second training set.

[0216] The second confirmation unit 604 is specifically used to confirm the position of the center of each image category on the hypersphere based on the features of the sample images corresponding to each image category in the first training set; and confirm the image categories corresponding to the at least two target positions based on the position of the center of each image category on the hypersphere.

[0217] The second confirmation unit 604 is specifically configured to perform the following operations on the features of the sample images corresponding to each image category:

[0218] Determine an average value of features of the sample images corresponding to the first image category as a correction value;

[0219] determining a predicted position of a center of the first image category on the hypersphere based on a sum of features of sample images corresponding to the first image category and a Euclidean distance between features of the sample images corresponding to the first image category;

[0220] Based on the predicted position and the correction value, the position of the center of the first image class on the hypersphere is determined.

[0221] The second confirmation unit 604 is specifically configured to confirm that the image category of the first target position among the at least two target positions is the image category corresponding to the center of the hypersphere that is shortest in distance to the first target position.

[0222] The loss determination unit 605 is specifically used to determine the loss function of the first encoder based on the characteristics of the first sample image, the characteristics of each sample image included in the second training set, the characteristics of other sample images other than the first sample image in the first training set, the characteristics of at least one second sample image in the first training set that belongs to the same image category as the first sample image, the at least two target positions, and the target position corresponding to the image category of the first sample image.

[0223] The loss determining unit 605 is specifically configured to determine the standard comparison loss value based on the features of the first sample image, the features of each sample image included in the second training set, the features of at least one second sample image, the number of the at least one second sample image, the features of other sample images in the first training set except the first sample image, and at least two target positions;

[0224] Determining a position contrast loss value based on a feature of the first sample image, a target position corresponding to the image category of the first sample image, features of each sample image included in the second training set, features of other sample images in the first training set except the first sample image, and at least two target positions;

[0225] A loss function of the first encoder is determined based on the number of sample images in the first training set, the standard contrast loss value, and the position contrast loss value.

[0226] The adjustment unit 606 is specifically used to adjust the parameters of the first encoder based on the loss function of the encoder; retrain the first encoder with adjusted parameters based on the first training set and determine the loss function until the loss function meets the first condition, then confirm that the training of the first encoder is completed and it is the second encoder.

[0227] The iterative unit 607 is specifically used to determine at least three target positions on the hypersphere based on the number of image categories included in the third training set; perform image enhancement on the third sample image in the third training set to generate a fifth training set; confirm the features corresponding to each sample image in the third training set and the features corresponding to each sample image in the fifth training set based on the second encoder included in the image classification model; confirm the image categories corresponding to the at least three target positions based on the features of the sample images corresponding to each image category in the third training set; determine the loss function of the second encoder based on the features of each sample image included in the third training set, the features of each sample image included in the fifth training set, and the image categories corresponding to the at least three target positions; adjust the parameters of the second encoder based on the loss function of the second encoder, and confirm that the second encoder with adjusted parameters is the third encoder.

[0228] In some embodiments, the second image category and the third image category in the fourth training set are the fourth image category in the first training set; the fifth image category, the sixth image category and the seventh image category in the fourth training set are the eighth image category in the first training set; and the ninth image category and the tenth image category in the fourth training set are the eleventh image category in the third training set.

[0229] Figure 7 An optional structural diagram of an image classification device provided by an embodiment of the present disclosure is shown, and the following description will be given according to each step.

[0230] In some embodiments, the image classification apparatus 700 includes a first input unit 701 and a second input unit 702 .

[0231] The first input unit 701 is used to input the image to be classified into the encoder included in the image classification model, confirm that the output of the encoder is the feature of the image to be classified, and the feature of the image to be classified is reflected in the hypersphere.

[0232] The second input unit 702 is used to input the features of the image to be classified into the linear classification layer included in the image classification model, and confirm that the output of the linear classification layer is the image category of the image to be classified; wherein the encoder includes a third encoder or a fourth encoder.

[0233] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0234] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0235] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0236] Multiple components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0237] The computing unit 801 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the image classification model training method and / or the image classification method. For example, in some embodiments, the image classification model training method and / or the image classification method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the image classification model training method and / or the image classification method described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the image classification model training method and / or the image classification method in any other appropriate manner (for example, by means of firmware).

[0238] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0239] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0240] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0241] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0242] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0243] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0244] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0245] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0246] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A training method for an image classification model, characterized in that: The method comprises: determining at least two target locations on the hypersphere based on the number of image categories included in the first training set; Performing image enhancement on the first sample image in the first training set to generate a second training set; Determining, based on a first encoder included in the image classification model, features corresponding to each sample image in the first training set and features corresponding to each sample image in the second training set; Determining the image categories corresponding to the at least two target positions based on features of sample images corresponding to each image category in the first training set; determining a loss function of the first encoder based on features of each sample image included in the first training set, features of each sample image included in the second training set, and image categories corresponding to the at least two target positions; adjusting parameters of the first encoder based on the loss function of the first encoder, and confirming that the first encoder with the adjusted parameters is the second encoder; Inputting the third training set into the second encoder to train the second encoder, and confirming that the trained second encoder is the third encoder; The third training set includes or excludes the first training set, and the number of image categories in the third training set is greater than the number of image categories in the first training set.

2. The method according to claim 1, characterized in that The method further comprises: Inputting the fourth training set into the third encoder to train the third encoder, and confirming that the third encoder is the fourth encoder after the training; The fourth training set includes the first training set and the third training set, and the number of image categories in the fourth training set is greater than the number of image categories in the third training set.

3. The method according to claim 2, characterized in that The method further comprises: Inputting the third training set into the third encoder and the linear classification layer included in the image classification model, and determining that the output of the linear classification layer is the predicted image category corresponding to each sample image in the third training set; Adjusting the parameters of the linear classification layer based on the image category corresponding to each sample image in the third training set and the predicted image category; Alternatively, the fourth training set is input into the fourth encoder and a linear classification layer included in the image classification model, and an output of the linear classification layer is determined to be a predicted image category corresponding to each sample image in the fourth training set; Based on the image category corresponding to each sample image in the fourth training set and the predicted image category, the parameters of the linear classification layer are adjusted.

4. The method according to claim 1, wherein The step of determining at least two target positions on the hypersphere based on the number of image categories included in the first training set comprises: At least two target positions on the hypersphere are determined based on the number of image categories included in the first training set and a minimization formula.

5. The method according to claim 1, wherein The performing image enhancement on the first sample image in the first training set to generate a second training set includes: Perform at least one of random cropping, rotation, random blocking, and color change on the first sample image to generate at least two non-identical images as sample images in the second training set.

6. The method according to claim 1, characterized in that The first encoder included in the image classification model confirms the features corresponding to each sample image in the first training set and the features corresponding to each sample image in the second training set, including: Inputting each sample image included in the first training set and the image category corresponding to each sample image included in the first training set into a first encoder included in the image classification model, and outputting features of each sample image included in the first training set; The sample images included in the second training set and the image categories corresponding to the sample images included in the second training set are input into the first encoder, and the features of the sample images included in the second training set are output.

7. The method according to claim 1, characterized in that The determining, based on features of sample images corresponding to each image category in the first training set, the image categories corresponding to the at least two target positions includes: Determining the position of the center of each image category on the hypersphere based on features of the sample images corresponding to each image category in the first training set; The image categories corresponding to the at least two target positions are determined based on the position of the center of each image category on the hypersphere.

8. The method according to claim 7, characterized in that Determining the position of the center of each image category on the hypersphere based on the features of the sample images corresponding to each image category in the first training set includes performing the following operations on the features of the sample images corresponding to each image category: Determine an average value of features of the sample images corresponding to the first image category as a correction value; determining a predicted position of a center of the first image category on the hypersphere based on a sum of features of sample images corresponding to the first image category and a Euclidean distance between features of the sample images corresponding to the first image category; Based on the predicted position and the correction value, the position of the center of the first image class on the hypersphere is determined.

9. The method according to claim 1 or 8, characterized in that The determining, based on the position of the center of each image category on the hypersphere, the image categories corresponding to the at least two target positions includes: It is determined that the image category of a first target position among the at least two target positions is an image category corresponding to a center on the hypersphere that is shortest in distance from the first target position.

10. The method according to claim 1, characterized in that The determining the loss function of the first encoder based on the features of each sample image included in the first training set, the features of each sample image included in the second training set, and the image categories corresponding to the at least two target positions includes: Determine the loss function of the first encoder based on the features of the first sample image, the features of each sample image included in the second training set, the features of other sample images in the first training set except the first sample image, the features of at least one second sample image in the first training set that belongs to the same image category as the first sample image, the at least two target positions, and the target position corresponding to the image category of the first sample image.

11. The method according to claim 10, characterized in that The determining the loss function of the first encoder based on the features of each sample image included in the first training set, the features of each sample image included in the second training set, and the image categories corresponding to the at least two target positions includes: confirming standard contrast loss values based on a feature of the first sample image, features of each sample image included in the second training set, features of at least one second sample image, the number of the at least one second sample image, features of other sample images in the first training set except the first sample image, and at least two target positions; Determining a position contrast loss value based on a feature of the first sample image, a target position corresponding to the image category of the first sample image, features of each sample image included in the second training set, features of other sample images in the first training set except the first sample image, and at least two target positions; A loss function of the first encoder is determined based on the number of sample images in the first training set, the standard contrast loss value, and the position contrast loss value.

12. The method according to claim 1, characterized in that The adjusting the parameters of the first encoder based on the loss function of the first encoder, and confirming that the first encoder with the adjusted parameters is the second encoder, includes: Adjusting parameters of the first encoder based on a loss function of the encoder; The first encoder after parameter adjustment is retrained based on the first training set, and a loss function is determined until the loss function satisfies a first condition, and the training of the first encoder is confirmed to be completed, and the encoder becomes the second encoder.

13. The method according to claim 1, wherein Inputting the third training set into the second encoder, training the second encoder, and confirming that the trained second encoder is the third encoder includes: determining at least three target locations on the hypersphere based on the number of image categories included in the third training set; performing image enhancement on the third sample image in the third training set to generate a fifth training set; confirming, based on the second encoder included in the image classification model, features corresponding to each sample image in the third training set and features corresponding to each sample image in the fifth training set; Determining the image categories corresponding to the at least three target positions based on features of the sample images corresponding to each image category in the third training set; determining a loss function of the second encoder based on features of each sample image included in the third training set, features of each sample image included in the fifth training set, and image categories corresponding to the at least three target positions; Based on the loss function of the second encoder, the parameters of the second encoder are adjusted, and the second encoder with the adjusted parameters is confirmed to be the third encoder.

14. The method according to claim 2, characterized in that The second image category and the third image category in the fourth training set are the fourth image category in the first training set; the fifth image category, the sixth image category and the seventh image category in the fourth training set are the eighth image category in the first training set; In the fourth training set, the ninth image category and the tenth image category are the eleventh image category in the third training set.

15. An image classification method, characterized in that: Applying an image classification model trained based on any one of claims 1 to 14, the method comprises: Inputting the image to be classified into an encoder included in the image classification model, and confirming that the output of the encoder is a feature of the image to be classified; the feature of the image to be classified is reflected in a hypersphere; Inputting the features of the image to be classified into a linear classification layer included in the image classification model, and confirming that the output of the linear classification layer is the image category of the image to be classified; Wherein, the encoder includes a third encoder or a fourth encoder.

16. A training device for an image classification model, characterized in that: The device comprises: a target position determining unit, configured to determine at least two target positions on the hypersphere based on the number of image categories included in the first training set; a training set enhancement unit, configured to perform image enhancement on the first sample image in the first training set to generate a second training set; a first confirmation unit, configured to confirm, based on a first encoder included in the image classification model, features corresponding to each sample image in the first training set and features corresponding to each sample image in the second training set; a second confirmation unit, configured to confirm the image categories corresponding to the at least two target positions based on features of sample images corresponding to each image category in the first training set; a loss determining unit, configured to determine a loss function of the first encoder based on features of each sample image included in the first training set, features of each sample image included in the second training set, and image categories corresponding to the at least two target positions; an adjusting unit, configured to adjust parameters of the first encoder based on a loss function of the first encoder, and confirm that the first encoder whose parameters are adjusted is a second encoder; an iterative unit, configured to input the third training set into the second encoder, train the second encoder, and confirm that the trained second encoder is a third encoder; The third training set includes or excludes the first training set, and the number of image categories in the third training set is greater than the number of image categories in the first training set.

17. An image classification device, characterized in that: Applying an image classification model trained based on any one of claims 1 to 14, the device comprises: A first input unit is configured to input an image to be classified into an encoder included in the image classification model, and confirm that an output of the encoder is a feature of the image to be classified, wherein the feature of the image to be classified is reflected in a hypersphere; A second input unit is configured to input the features of the image to be classified into a linear classification layer included in the image classification model, and confirm that the output of the linear classification layer is the image category of the image to be classified; Wherein, the encoder includes a third encoder or a fourth encoder.

18. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14; Or perform the method according to claim 15.

19. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14; Or perform the method according to claim 15.

Images