Model training method, image classification method and device

Abstract

The embodiments of the present application provide a model training method, an image classification method, and an apparatus, relating to the field of computer vision and pattern recognition technology. The method comprises: obtaining a plurality of first sample images; performing at least one training operation on an initial classification model based on the plurality of first sample images, and using the initial classification model that meets the preset training end conditions as a trained classification model; the training operation comprises: determining the prediction region where the sample target object in each first sample image is located; for each first sample image, segmenting the first sample image according to the determined prediction region, and obtaining at least one second sample image corresponding to the first sample image; determining the prediction category corresponding to the at least one second sample image, and the loss function; and adjusting the parameters of the initial classification model based on the loss function. The embodiments of the present application improve the accuracy of the classification model obtained through training.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and pattern recognition. Specifically, the present application relates to a model training method, an image classification method and a device. Background Art

[0002] With the development of machine learning, neural network models have become increasingly widely used. During the training phase, neural network models often require a large amount of training data to achieve better performance. However, in practical applications, large amounts of training data may not be available. To address this situation, small-sample learning methods have emerged.

[0003] In image classification applications, existing small-sample learning methods typically expand the number of samples by gridding existing sample images. However, when some sample target objects are small or located in a local area of ​​the corresponding image, the image obtained after gridding contains the background of the image but does not contain the sample target object. This results in the actual content of these images being inconsistent with their corresponding labels, and in turn, the accuracy of the model trained using these images is low. Summary of the Invention

[0004] The embodiments of the present application provide a model training method, an image classification method and an apparatus, which can solve the problem of low accuracy of the trained model in the prior art.

[0005] The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a model training method is provided, the method comprising:

[0007] acquiring a plurality of first sample images;

[0008] Performing at least one training operation on the initial classification model based on the multiple first sample images, and using the initial classification model that meets the preset training end condition as the trained classification model;

[0009] The training operation includes:

[0010] Determining a predicted region where a sample target object is located in each first sample image;

[0011] For each first sample image, segment the first sample image according to the determined prediction region to obtain at least one second sample image corresponding to the first sample image; the second sample image includes at least a portion of the prediction region where the sample target object is located;

[0012] Determining the predicted categories corresponding to the at least one second sample image, and determining a loss function based on the true categories corresponding to the respective sample target objects and the predicted categories corresponding to the respective second sample images;

[0013] The parameters of the initial classification model are adjusted based on the loss function, and the initial classification model after the parameters are adjusted is used as the initial classification model corresponding to the next training operation.

[0014] Optionally, for each first sample image, segmenting the first sample image according to the determined prediction area to obtain at least one second sample image corresponding to the first sample image includes:

[0015] determining at least one candidate region of the first sample image based on at least one detection box corresponding to the predicted region where the sample target object is located in the first sample image;

[0016] Based on the determined at least one candidate area, at least one corresponding second sample image is generated respectively.

[0017] Optionally, the candidate area includes a first candidate area and a second candidate area;

[0018] The determining, based on at least one detection frame corresponding to the predicted region where the sample target object is located in the first sample image, at least one candidate region of the first sample image includes:

[0019] Determine a first candidate region corresponding to each detection frame;

[0020] Determine the minimum bounding rectangle area of ​​each detection frame to obtain the second candidate area.

[0021] Optionally, generating at least one corresponding second sample image based on the determined at least one candidate region includes:

[0022] Based on the at least one candidate region, obtaining at least one third sample image corresponding to the at least one candidate region;

[0023] Random data enhancement processing is performed on the at least one third sample image to obtain at least one second sample image.

[0024] Optionally, the random data enhancement process includes at least one of the following:

[0025] Random erase processing;

[0026] Random cropping process;

[0027] Random rotation processing;

[0028] Random noise processing.

[0029] Optionally, the plurality of first sample images include a plurality of query images, a positive example image having the same category as each query image, and a negative example image having a different category from each query image.

[0030] The determining of the loss function based on the true category corresponding to each sample target object and the predicted category corresponding to each second sample image includes:

[0031] Determining the label category corresponding to each second sample image based on the true category corresponding to each sample target object;

[0032] Determining a first loss function based on first differences between label categories corresponding to respective second sample images and their predicted categories;

[0033] determining a second loss function based on a second difference between any two second sample images belonging to the same category and a third difference between any two second sample images not belonging to the same category;

[0034] The loss function is determined based on the first loss function and the second loss function.

[0035] According to one aspect of an embodiment of the present application, a method for image classification is provided, the method comprising:

[0036] Acquire a first image to be classified;

[0037] The trained classification model is used to perform classification operations to obtain the image category of the first image:

[0038] The classification operation includes:

[0039] Performing target detection on the first image to obtain a target area where a target object in the first image is located;

[0040] Segmenting the first image based on a target region where a target object is located in the first image to obtain at least one second image corresponding to the first image; the second image includes at least a portion of the target region where the target object is located;

[0041] classifying the at least one second image to obtain image categories corresponding to the respective second images, and determining the image category of the first image based on the image categories corresponding to the respective second images;

[0042] The classification model is obtained by training based on the model training method provided in any optional embodiment of the present application.

[0043] According to another aspect of an embodiment of the present application, a model training device is provided, the device comprising:

[0044] A first sample image acquisition module, configured to acquire a plurality of first sample images;

[0045] a training module, configured to perform at least one training operation on the initial classification model based on the plurality of first sample images, and use the initial classification model that meets a preset training end condition as a trained classification model;

[0046] The training operation includes:

[0047] Determining a predicted region where a sample target object is located in each first sample image;

[0048] For each first sample image, segment the first sample image according to the determined prediction region to obtain at least one second sample image corresponding to the first sample image; the second sample image includes at least a portion of the prediction region where the sample target object is located;

[0049] Determining the predicted categories corresponding to the at least one second sample image, and determining a loss function based on the true categories corresponding to the respective sample target objects and the predicted categories corresponding to the respective second sample images;

[0050] The parameters of the initial classification model are adjusted based on the loss function, and the initial classification model after the parameters are adjusted is used as the initial classification model corresponding to the next training operation.

[0051] According to another aspect of an embodiment of the present application, there is provided an image classification apparatus, the apparatus comprising:

[0052] A first image acquisition module, configured to acquire a first image to be classified;

[0053] A classification module is configured to perform a classification operation using a trained classification model to obtain an image category of the first image:

[0054] The classification operation includes:

[0055] Performing target detection on the first image to obtain a target area where a target object in the first image is located;

[0056] Segmenting the first image based on a target region where a target object is located in the first image to obtain at least one second image corresponding to the first image; the second image includes at least a portion of the target region where the target object is located;

[0057] classifying the at least one second image to obtain image categories corresponding to the respective second images, and determining the image category of the first image based on the image categories corresponding to the respective second images;

[0058] The classification model is obtained by training based on the model training method provided in any optional embodiment of the present application.

[0059] According to another aspect of an embodiment of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor, and when the processor executes the program, the steps of any one of the above-mentioned model training methods or image classification methods are implemented.

[0060] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model training methods or image classification methods are implemented.

[0061] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0062] For each first sample image, target detection is performed on the first sample image to obtain a predicted region where the sample target object is located. The first sample image is then segmented based on the predicted region, so that at least one second sample image generated contains at least a portion of the corresponding sample target object. This not only achieves the goal of expanding the number of samples by segmenting the original first sample image, but also ensures that the expanded second sample images all contain real target objects. This further ensures that the actual content of the second sample image is consistent with its label category, allowing for subsequent effective training of the initial classification model using the second sample images, thereby improving the accuracy of the trained classification model.

[0063] In addition, the predicted area where the sample target object is located is used as the basis for dividing the first sample image, so that the second sample images obtained by division are all related to the sample target object, thereby eliminating irrelevant background areas, reducing the interference of background areas in the image, and facilitating the rapid convergence of the model and improving the training speed of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0065] Figure 1 A schematic diagram of grid division of a query image in the prior art;

[0066] Figure 2A flowchart of a model training method provided in an embodiment of the present application;

[0067] Figure 3 A flowchart of another model training method provided in an embodiment of the present application;

[0068] Figure 4 A schematic diagram of an RPN module structure provided in an embodiment of the present application;

[0069] Figure 5 A flowchart of an image classification method provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of the structure of an image classification device provided in an embodiment of the present application;

[0072] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0074] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to that the element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0075] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0076] Neural network models often require a large amount of training data during the training phase to achieve better performance. However, in real-world applications, this large amount of training data may not be available. To address this situation, small-sample learning methods have emerged. Small-sample learning methods aim to effectively perform machine learning with a small number of training samples.

[0077] In image classification applications, it is sometimes difficult to collect enough data to train the model. Therefore, small sample learning can be used to perform image classification to solve the problem of small training data volume. Existing small sample image classification methods include the following:

[0078] 1. Model-based improvement method

[0079] The core idea of ​​this method is to minimize the hypothesis space. This method can be further categorized into the following subcategories: Multi-task learning methods: These methods can handle multiple tasks, where model parameters can be shared or correlated; Embedded learning methods: These methods map samples to a low-dimensional space, thereby reducing the hypothesis space and approximating the model solution using a small number of samples; and External memory-based learning methods: These methods store the knowledge learned from a small sample dataset externally, and then use this stored knowledge to represent new samples and perform matching based on this representation.

[0080] 2. Algorithm optimization-based methods

[0081] The core of this method is to search for a suitable solution faster by improving the optimization algorithm. This method can be divided into the following subcategories. Methods for improving existing parameters: Starting from the perspective of parameter initialization, the parameters of the small sample model are adjusted with the help of the trained model. Methods for improving meta-learning parameters: The idea of ​​this method is to learn how to learn, and to initialize the parameters of the new task by combining the parameters of different specific task models. Methods for learning optimizers: The optimizer iteratively updates the parameters of the previous model each time, and applies the updated values ​​of each iteration in the learning small sample data set to the new test set.

[0082] At present, the most common small-sample image classification method is the small-sample embedding model method with contrastive learning (infoPatch for short). The small-sample embedding model with contrastive learning refers to learning embedded representations through contrastive learning in the case of small samples. It belongs to the embedding learning method among the above-mentioned model improvement-based methods.

[0083] The existing infoPatch method divides the query image in the sample image into N*W grid area images, where each grid area image is a patch (small block) and all grid area images are a group of patches.

[0084] Figure 1 A schematic diagram of grid division of a query image in the prior art is shown as follows: Figure 1 As shown in Figure 3, when some sample target objects are small or located in a local area of ​​the corresponding image, when the query image is divided into patches using a grid, some of the generated patches may be just backgrounds and do not contain real objects corresponding to the real categories.

[0085] The category label of each patch in a set of Patches corresponding to the query image is the same as the category label of the original query image, which is the true category of the target object in the query image. For example, Figure 1 If the query image in the image has a category label of bird, then each patch in the set of patches corresponding to the query image will have a category label of bird. However, if the bird in the query image is small, some of the patches after the network partitioning will not contain birds. As a result, these patches will not contain the objects corresponding to their category labels. The actual content of these patches is inconsistent with their corresponding category labels, that is, the category labels of these patches are not their true categories.

[0086] The model training process involves continuously adjusting the model parameters to ensure that the predicted category of the sample image output by the model is as consistent as possible with the category label. Because the actual content of these patches is inconsistent with their corresponding category labels, the model trained on these patches has low accuracy and cannot effectively classify images.

[0087] The model training method, image classification method and device provided in this application are intended to solve the above technical problems of the prior art.

[0088] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0089] Figure 2 A flow chart of a model training method provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method includes:

[0090] Step S110: Acquire a plurality of first sample images.

[0091] Specifically, the first sample image can be an image used for model training. The first sample image can be collected by an image acquisition device, or it can be collected through the network under the premise of complying with relevant regulations. The embodiment of the present application does not limit the specific method of obtaining the first sample image.

[0092] For each first sample image, the first sample image may be a labeled sample image, the first sample image has a corresponding label category, and the label category of the first sample image may be a true category of at least one sample target object contained in the first sample image.

[0093] Step S120 , performing at least one training operation on the initial classification model based on the plurality of first sample images, and taking the initial classification model that meets the preset training end condition as the trained classification model.

[0094] Specifically, the training operation is repeatedly performed on the initial classification model based on the plurality of first sample images until a preset training end condition is met.

[0095] The training operation includes the following steps:

[0096] Step S121, determining a prediction region where a sample target object is located in each first sample image;

[0097] Step S122: for each first sample image, segment the first sample image according to the determined prediction region to obtain at least one second sample image corresponding to the first sample image; the second sample image includes at least a portion of the prediction region where the sample target object is located;

[0098] Step S123, determining the predicted category corresponding to at least one second sample image, and determining a loss function based on the true category corresponding to each sample target object and the predicted category corresponding to each second sample image;

[0099] Step S124 , adjusting the parameters of the initial classification model based on the loss function, and using the initial classification model after adjusting the parameters as the initial classification model corresponding to the next training operation.

[0100] Specifically, a plurality of first sample images are input into an initial classification model, and target detection is performed on the plurality of first sample images by the initial classification model to obtain a predicted region where a sample target object in each first sample image is located.

[0101] For each first sample image, the first sample image may be segmented based on the predicted region where the sample target object is located in the first sample image, and at least one corresponding second sample image may be generated based on at least one image obtained after the segmentation. The second sample image may include at least a portion of the predicted region where the sample target object is located in the corresponding first sample image.

[0102] The initial classification model may include an initial detection module, an initial image segmentation module, and an initial category prediction module. The initial detection module is used to perform target detection on the first sample image and output a predicted region where the sample target object is located in the first sample image; the initial image segmentation module is used to segment the first sample image based on the predicted region; and the initial category prediction module is used to perform image classification on the second sample image and output a predicted category for the second sample image.

[0103] The initial detection module and the initial category prediction module can be constructed based on a neural network. Optionally, the initial detection module and the initial category prediction module can share part of the neural network structure, that is, the initial detection module and the initial category prediction module have part of the same neural network structure. This shared neural network structure can share model parameters, thereby reducing the amount of model parameter calculation and improving the speed of model training.

[0104] Optionally, at least one image obtained by segmenting the first sample image based on the prediction area can be used as the second sample image; or at least one image obtained after segmentation can be combined, and the combined image can be used as the second sample image. The embodiment of the present application does not limit the specific method for determining the second sample image.

[0105] Compared to the network segmentation methods used in the prior art, the method provided in the embodiments of the present application specifically segments the first sample image based on the predicted region where the sample target object is located, ensuring that the generated second sample image contains at least a portion of the corresponding sample target object. This method not only achieves the goal of expanding the sample quantity by segmenting the original first sample image, but also ensures that the expanded second sample images contain the actual sample target object, thereby ensuring the consistency between the actual content of the second sample image and its label category, allowing for subsequent effective training of the initial classification model using the second sample image, thereby improving the accuracy of the trained classification model.

[0106] After determining at least one second sample image, the initial classification model can be used to obtain a predicted category corresponding to each second sample image. For each second sample image, the true category of the sample target object contained in the second sample image can be used as the label category of the second sample image. Based on the difference between the label category corresponding to each second sample image and the predicted category, a loss function of the initial classification model is determined.

[0107] Based on the loss function, the parameters of the initial classification model corresponding to the current training operation can be adjusted, and the initial classification model with adjusted parameters can be used in the next training operation. By continuously executing the above training operation, the training of the model is constrained based on the loss function, so that the predicted category of the initial classification model is closer and closer to the label category of the sample image, until the preset training end conditions are met. The initial classification model that meets the preset training end conditions is then regarded as the trained classification model.

[0108] Among them, the training end condition can be that the loss function converges, for example, the loss function is less than the set value or the loss function calculated a set number of times is less than the set value; the training end condition can also be that the number of training times reaches a preset number, which is not limited in this embodiment of the present application.

[0109] The model training method provided in the embodiments of the present application performs target detection on each first sample image to obtain a predicted region where a sample target object is located in the first sample image. The first sample image is then segmented based on the predicted region, so that at least one generated second sample image contains at least a portion of the corresponding sample target object. This method not only achieves the purpose of expanding the number of samples by segmenting the original first sample image, but also ensures that the expanded second sample images contain real target objects, thereby ensuring the consistency between the actual content of the second sample image and its label category, thereby facilitating the subsequent effective training of the initial classification model using the second sample images, thereby improving the accuracy of the trained classification model.

[0110] In addition, when the sample target object in the first sample image is small, the corresponding background area in the first sample image is large. The background area is usually information irrelevant to the sample target object. When an image with a large background area is used as a training sample, it will cause significant interference to the model training, which is not conducive to the rapid convergence of the model and reduces the training speed of the model. In the embodiment of the present application, the prediction area where the sample target object is located is used as the basis for dividing the first sample image, so that the second sample images obtained by the division are all related to the sample target object, thereby eliminating irrelevant background areas, reducing the interference of the background area in the image, facilitating the rapid convergence of the model, and improving the training speed of the model.

[0111] As an optional embodiment, for each first sample image, segmenting the first sample image according to the determined prediction region to obtain at least one second sample image corresponding to the first sample image includes:

[0112] determining at least one candidate region of the first sample image based on at least one detection frame corresponding to the predicted region where the sample target object is located in the first sample image;

[0113] Based on the determined at least one candidate region, at least one corresponding second sample image is generated respectively.

[0114] Specifically, for each first sample image, at least one corresponding candidate region may be determined based on at least a portion of the predicted region where the sample target object in the first sample image is located.

[0115] The predicted region where the sample target object in the first sample image is located can be represented as multiple detection frames, and the region corresponding to any detection frame can be used as a candidate region; alternatively, the region containing at least two detection frames can be used as a candidate region. Furthermore, the region containing at least two detection frames can be determined based on a minimum bounding rectangle containing the at least two detection frames.

[0116] For example, the predicted area is represented by four detection frames, that is, the predicted area corresponds to four detection frames. The area containing any two detection frames of the four detection frames can be used as a candidate area, the area containing any three detection frames of the four detection frames can be used as a candidate area, and the area containing all four detection frames can be used as a candidate area.

[0117] After determining at least one candidate region corresponding to the first sample image, at least one corresponding second sample image may be generated based on the image corresponding to the at least one candidate region.

[0118] In the embodiment of the present application, at least one candidate region is determined based on at least one detection frame corresponding to the predicted region, and at least one second sample image is generated based on the at least one candidate region. Because the detection frame at least partially includes the corresponding sample target object, the candidate region determined based on the at least one detection frame may also at least partially include the corresponding sample target object, and thus the second sample image generated based on the candidate region may also at least partially include the corresponding sample target object.

[0119] At the same time, the predicted region is represented as at least one detection box, and multiple candidate regions are generated by arbitrarily combining at least one detection box corresponding to the predicted region, thereby further expanding the number of training samples.

[0120] As an optional embodiment, the candidate area includes a first candidate area and a second candidate area;

[0121] Determining at least one candidate region of the first sample image based on at least one detection frame corresponding to a predicted region where a sample target object is located in the first sample image includes:

[0122] Determine a first candidate region corresponding to each detection frame;

[0123] Determine the minimum bounding rectangle area of ​​each detection frame to obtain the second candidate area.

[0124] Specifically, the candidate area may include a first candidate area and a second candidate area. The area corresponding to each detection frame corresponding to the prediction area may be used as the first candidate area, and the minimum circumscribed rectangular area containing each detection frame may be used as the second candidate area.

[0125] Each detection frame can at least partially include the corresponding sample target object, that is, the first candidate region corresponding to each detection frame can include at least a portion of the sample target object, but there is no guarantee that the first candidate region can include the entire sample target object. By using the minimum circumscribed rectangular region containing all detection frames as the second candidate region, the second candidate region can include the entire sample target object, so that the second sample image generated based on the second candidate region can fully reflect the characteristics of the corresponding sample target object, allowing the sample image to better represent the corresponding real category, which is conducive to improving the accuracy of the trained model.

[0126] As an optional embodiment, generating at least one corresponding second sample image based on the determined at least one candidate region includes:

[0127] Based on the at least one candidate region, obtaining at least one third sample image corresponding to the at least one candidate region;

[0128] Random data enhancement processing is performed on the at least one third sample image to obtain at least one second sample image.

[0129] Specifically, for each first sample image, after determining at least one candidate region corresponding to the first sample image, an image corresponding to the at least one candidate region may be used as the corresponding at least one third sample image.

[0130] Several candidate sample images can be selected from at least one third sample image, wherein some of the third sample images can be used as candidate sample images, or all of the third sample images can be used as candidate sample images, which is not limited in this embodiment of the present application.

[0131] For any candidate sample image, random data enhancement processing is performed on the candidate sample image, and the candidate sample image after the random data enhancement processing is used as the second sample image.

[0132] The random data enhancement processing may include at least one of random erasing processing, random cropping processing, random rotation processing and random noise processing.

[0133] In an embodiment of the present application, by adjusting at least one of the number of data enhancement processes in the random data enhancement process, the method of each data enhancement process, and the order of each data enhancement process, the number of training samples can be further expanded and the diversity of training samples can be improved by enriching the specific methods of data enhancement.

[0134] Alternatively, different methods can be used to perform data augmentation on the same candidate sample image. For example, random data augmentation can be performed twice on any candidate sample image to obtain a second sample image; random data augmentation can be performed three times on the same candidate sample image to obtain another second sample image. For another example, random erasing can be performed once on any candidate sample image to obtain a second sample image; random cropping can be performed once on the same candidate sample image to obtain another second sample image.

[0135] Optionally, for different candidate sample images, data enhancement may be performed in the same manner or in different manners.

[0136] In the embodiment of the present application, random data augmentation processing is performed on the third sample image to obtain the second sample image, which not only expands the number of training samples but also improves the diversity of training samples, thereby improving the generalization ability of the model.

[0137] As an optional embodiment, the plurality of first sample images include a plurality of query images, at least one positive example image of the same category as each query image, and at least one negative example image of a different category than each query image.

[0138] Based on the true category corresponding to each sample target object and the predicted category corresponding to each second sample image, a loss function is determined, including:

[0139] Determining the label category corresponding to each second sample image based on the true category corresponding to each sample target object;

[0140] Determining a first loss function based on first differences between label categories corresponding to respective second sample images and their predicted categories;

[0141] determining a second loss function based on a second difference between any two second sample images belonging to the same category and a third difference between any two second sample images not belonging to the same category;

[0142] A loss function is determined based on the first loss function and the second loss function.

[0143] Specifically, the multiple first sample images may include multiple query images, multiple positive example images, and multiple negative example images, wherein the positive example images may be images of the same category as the query images, and the negative example images may be images of a different category from the query images.

[0144] For each first sample image, target detection can be performed on the first sample image using the initial classification model to obtain a predicted region where the sample target object is located in the first sample image. The first sample image can then be segmented based on the predicted region to obtain at least one corresponding second sample image. The label category of the second sample image is the same as the label category of the corresponding first sample image.

[0145] The above segmentation operation can be performed on the query image, positive image, and negative image in the first sample image to generate at least one corresponding second sample image. This expands the number of training samples and simultaneously removes irrelevant background areas, reducing interference from background areas in the image, facilitating rapid model convergence and improving model training speed. Furthermore, the original first sample image can be used as a second sample image to further expand the number of training samples.

[0146] On this basis, the loss function of model training may include a first loss function and a second loss function.

[0147] For each second sample image, the true category of the sample target object contained in the second sample image can be used as the label category of the second sample image. A first loss function is determined based on the first difference between the label category corresponding to each second sample image and the predicted category.

[0148] For each second sample image, a second loss function may also be determined based on a second difference between image features of any two second sample images belonging to the same category, and a third difference between any two second sample images not belonging to the same category.

[0149] A loss function for constrained model training is determined based on the first loss function and the second loss function. The loss function of the model can be obtained by summing the first loss function and the second loss function. Optionally, weights corresponding to the first loss function and the second loss function can be obtained, and the weighted sum of the two losses can be used to obtain the loss function of the model.

[0150] In an embodiment of the present application, the loss function of the model includes two parts: one part is the difference between the predicted category of each second sample image obtained by the model and the label category of each second sample image, that is, the first loss function; the other part is the difference between second sample images of the same category and the difference between second sample images of different categories, that is, the second loss function. The initial classification model is trained by a joint loss function including the first loss function and the second loss function, so that the model can learn that the smaller the difference between the predicted category and the label category, the smaller the difference between image features of the same category, and the larger the difference between image features of different categories, thereby improving the accuracy of the trained classification model and facilitating the accuracy of subsequent image classification.

[0151] As an optional embodiment, Figure 3 A flow chart of a model training method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the model training method includes:

[0152] A plurality of first sample images are determined, where the plurality of first sample images include a plurality of query images, a plurality of positive example images, and a plurality of negative example images, wherein the positive example images may be images of the same category as the query images, and the negative example images may be images of a different category from the query images. Figure 3 The category of the query image is dog, the category of the positive image is also dog, and the category of the negative image is bird.

[0153] In an embodiment of the present application, the predicted region where the sample target object is located in each first sample image can be represented as multiple detection boxes. A detection box extractor can be used to perform target detection on each first sample image and output the corresponding multiple detection boxes. The detection box extractor can include a random data augmentation processing module, a backbone feature extractor, and an RPN (Region Proposal Network) module.

[0154] The random data augmentation processing module can perform random data augmentation on each first sample image. The random data augmentation processing can include at least one of random erasing, random cropping, random rotation, and random noise processing to increase the diversity of training samples. Furthermore, each first sample image can be resized to a preset size, for example, to 224*224.

[0155] For the first sample image after random data augmentation, the backbone features of the first sample image can be extracted by the backbone feature extractor. When the size of the first sample image input to the backbone feature extractor is 224*224, the size of the backbone features of the first sample image output is 512*7*7.

[0156] Optionally, the network structure of the backbone feature extractor may be a Resnet18 (a deep convolutional neural network) network structure without the average pooling layer and the FC (Full Connect) layer.

[0157] The extracted backbone features are input into the RPN module, and multiple convolution operations and NMS (Non Maximum Suppression) processing are performed through the RPN module to obtain the target detection results, where the target detection results may include the positions of a preset number of detection boxes and the corresponding scores.

[0158] Optionally, Figure 4 A schematic diagram of an RPN module structure provided in an embodiment of the present application is shown as follows: Figure 4 As shown in the figure, the RPN module includes multiple convolution kernels. Multiple backbone features of size 512*7*7 are input into the RPN module. Multiple detection frames and their corresponding scores are obtained through multiple convolution operations of multiple convolution kernels. The multiple detection frames and their corresponding scores are processed by NMS, and a preset number of detection frames with the highest scores and their scores are output. For example, if the preset number is 4, the first four detection frames with the highest scores are taken as the Top4 detection frames, and the Top4 detection frames and their scores are output.

[0159] For each first sample image, after obtaining a preset number of detection frames for the first sample image, the image corresponding to the area of ​​each detection frame can be used as a second sample image. Based on this, the MBR (Minimum Bounding Rectangle) including each detection frame can also be obtained, and the image corresponding to the area of ​​the minimum bounding rectangle can be used as a second sample image. The preset number of detection frames and the MBR area both contain at least a portion of the actual target object in the corresponding image.

[0160] Optionally, when outputting the top 4 detection frames, the MBR of the top 4 detection frames can be determined based on the position of each detection frame in the top 4 detection frames. For each detection frame, x1, y1, x2, and y2 represent the horizontal and vertical coordinates of the top left corner and the bottom right corner of the detection frame, respectively. The minimum x1 value of all four detection frames is used as the MBR's x1, the minimum y1 value of all four detection frames is used as the MBR's y1, the maximum x2 value of all four detection frames is used as the MBR's x2, and the maximum y2 value of all four detection frames is used as the MBR's y2.

[0161] like Figure 3 As shown, a first sample image corresponds to 4 detection frames, and 4 second sample images are generated based on the 4 detection frames respectively. Based on an MBR including 4 detection frames, a second sample image is generated accordingly. That is to say, based on a first sample image, four corresponding second sample images can be generated, thereby achieving the purpose of expanding the number of training samples.

[0162] After obtaining multiple second sample images, random data enhancement processing can be performed on at least one second sample image, wherein the random data enhancement processing can include at least one of random erasing processing, random cropping processing, random rotation processing and random noise processing to improve the diversity of training samples and thereby improve the generalization ability of the model. Figure 3 , which shows a case where the random data enhancement process is a random erasure process.

[0163] For the second sample image after random data augmentation processing, the image features of the second sample image can be extracted by the average pooling feature extractor. Optionally, the network structure of the average pooling feature extractor can be the network structure of Resnet18 without the FC layer. In this case, the average pooling feature extractor has one more average pooling layer than the backbone feature extractor, and other network structures can share the parameters of the model. Moreover, since the average pooling feature extractor has one more average pooling layer, the output feature size can be uniform regardless of the size of each detection frame. Figure 4 Based on the RPN module shown, the image feature size output by the average pooling feature extractor is 512*1*1.

[0164] After obtaining the image features of each second sample image, the cosine similarity between the image features of each two second sample images can be calculated. For each two second sample images, the contrastive learning loss can be calculated based on the cosine similarity between the image features of each two second sample images and the positive and negative example relationship between the labels of the two second sample images according to the contrastive learning formula.

[0165] At the same time, the predicted category of each second sample image can also be calculated through the predicted category output module. Optionally, the predicted category output module can include an FC layer and a softmax (a classification function) layer. The output channel of the FC layer is the number of categories of all training samples plus 1, that is, the score of each second sample image belonging to each category is obtained. After passing through the softmax layer, the probability of belonging to each category of each second sample image is obtained. For each second sample image, the classification loss can be calculated according to the predicted classification probability of each second sample image and the label category of the second sample image according to the formula of cross entropy loss.

[0166] The total training loss of the model is obtained by directly summing the contrast loss and the classification loss. The model parameters of the backbone feature extractor and the average pooling feature extractor are continuously adjusted based on the total training loss, thereby continuously optimizing the model parameters. During model training, an end-to-end training approach can be used to effectively extract the most representative features of the image.

[0167] The model training method provided in the embodiment of the present application uses RPN to find a detection frame containing a real target object in a first sample image, and segments the first sample image through the detection frame to generate a second sample image to ensure that each second sample image at least partially contains the target object; for the query image, positive image, and negative image, RPN is used to obtain a preset number of detection frames and a minimum circumscribed rectangle containing all the preset number of detection frames, which are used for subsequent image feature extraction to reduce interference from background areas in the image, which is conducive to rapid convergence of the model; random data augmentation processing is performed on each second sample image to increase the diversity of training data and ultimately improve the generalization ability of the model.

[0168] Figure 5 A flow chart of an image classification method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the method includes:

[0169] Step S210, obtaining a first image to be classified;

[0170] Step S220, performing a classification operation using the trained classification model to obtain an image category of the first image;

[0171] The classification operation includes the following steps:

[0172] Step S221, performing target detection on the first image to obtain a target area where a target object in the first image is located;

[0173] Step S222, segmenting the first image based on the target region where the target object is located in the first image to obtain at least one second image corresponding to the first image; the second image includes at least a portion of the target region where the target object is located;

[0174] Step S223: classify the at least one second image to obtain an image category corresponding to the at least one second image, and determine the image category of the first image based on the image category corresponding to the at least one second image;

[0175] The classification model is obtained by training based on the model training method provided in any optional embodiment of the present application.

[0176] Specifically, before executing step S210, the model training method provided in the embodiment of the present application can be used to obtain a classification model. The model training method provided in the embodiment of the present application can obtain a classification model with higher accuracy when the number of training samples is limited, thereby improving the accuracy of image classification.

[0177] In actual implementation, a first image to be classified may be obtained first, wherein the first image may be an image that needs to be classified. The first image is input into a classification model, and the first image is classified by the classification model to obtain an image category of the first image.

[0178] The specific process of the classification model classifying the first image includes:

[0179] First, target detection is performed on the first image to obtain a target region where the target object is located in the first image. Second, the first image is segmented based on the target region where the target object is located in the first image to obtain at least one corresponding second image, wherein the second image contains at least a portion of the target region where the target object is located. Finally, image classification is performed on each second image to obtain the image category corresponding to the second image. The image category of the first image is then determined based on the image category corresponding to each second image.

[0180] Among them, when the image categories corresponding to multiple second images are the same, the unified image category of the multiple second images can be used as the image category of the first image; when the image categories corresponding to multiple second images are different, the image category with the highest probability of appearing in the multiple second images can be used as the image category of the first image.

[0181] In an embodiment of the present application, the second image is obtained based on the target area where the target object is located. The second image includes at least a part of the target object. The second image can effectively characterize the characteristics of the target object. By classifying the second image, the category of the target object can be accurately obtained, that is, the image category of the first image, thereby improving the accuracy of image classification.

[0182] The target area where the target object is located is used as the basis for dividing the first image, so that the second images obtained by division are all related to the target object, thereby eliminating irrelevant background areas, reducing the interference of background areas in the image, and helping to improve the speed of model operation and the efficiency of image classification.

[0183] In addition, determining the image category of the first image based on the image categories corresponding to each second image can effectively avoid the impact of a single misclassification of the model on the image classification result, further improving the accuracy of image classification.

[0184] Figure 6A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device includes:

[0185] A first sample image acquisition module 310 is configured to acquire a plurality of first sample images;

[0186] A training module 320 is configured to perform at least one training operation on the initial classification model based on the plurality of first sample images, and to use the initial classification model that meets a preset training end condition as a trained classification model;

[0187] The training operations include:

[0188] Determining a predicted region where a sample target object is located in each first sample image;

[0189] For each first sample image, segment the first sample image according to the determined prediction region to obtain at least one second sample image corresponding to the first sample image; the second sample image includes at least a portion of the prediction region where the sample target object is located;

[0190] Determining a predicted category corresponding to at least one second sample image, and determining a loss function based on the true category corresponding to each sample target object and the predicted category corresponding to each second sample image;

[0191] The parameters of the initial classification model are adjusted based on the loss function, and the initial classification model after adjusting the parameters is used as the initial classification model corresponding to the next training operation.

[0192] As an optional embodiment, the training module further includes an image segmentation submodule, and the image segmentation submodule includes:

[0193] a candidate region determining unit, configured to determine at least one candidate region of the first sample image based on at least one detection frame corresponding to the predicted region where the sample target object is located in the first sample image;

[0194] The second sample image generating unit is configured to generate at least one corresponding second sample image based on the determined at least one candidate area.

[0195] As an optional embodiment, the candidate area in the device includes a first candidate area and a second candidate area;

[0196] The candidate region determination unit is specifically used to:

[0197] Determine a first candidate region corresponding to each detection frame;

[0198] Determine the minimum bounding rectangle area of ​​each detection frame to obtain the second candidate area.

[0199] As an optional embodiment, the second sample image generating unit in the device is specifically configured to:

[0200] Based on the at least one candidate region, obtaining at least one third sample image corresponding to the at least one candidate region;

[0201] Random data enhancement processing is performed on the at least one third sample image to obtain at least one second sample image.

[0202] As an optional embodiment, the random data enhancement processing in the device includes at least one of the following:

[0203] Random erase processing;

[0204] Random cropping process;

[0205] Random rotation processing;

[0206] Random noise processing.

[0207] As an optional embodiment, the plurality of first sample images in the apparatus include a plurality of query images, at least one positive example image of the same category as each query image, and at least one negative example image of a different category than each query image.

[0208] The training module also includes a loss function calculation submodule for:

[0209] Determining the label category corresponding to each second sample image based on the true category corresponding to each sample target object;

[0210] Determining a first loss function based on first differences between label categories corresponding to respective second sample images and their predicted categories;

[0211] determining a second loss function based on a second difference between any two second sample images belonging to the same category and a third difference between any two second sample images not belonging to the same category;

[0212] A loss function is determined based on the first loss function and the second loss function.

[0213] Figure 7 A schematic diagram of the structure of an image classification device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device includes:

[0214] A first image acquisition module 410 is used to acquire a first image to be classified;

[0215] The classification module 420 is configured to perform a classification operation using the trained classification model to obtain the image category of the first image:

[0216] The classification operations include:

[0217] Performing target detection on the first image to obtain a target area where a target object in the first image is located;

[0218] Segmenting the first image based on a target region where the target object is located in the first image to obtain at least one second image corresponding to the first image; the second image includes at least a portion of the target region where the target object is located;

[0219] classifying at least one second image to obtain image categories corresponding to each second image, and determining the image category of the first image based on the image categories corresponding to each second image;

[0220] The classification model is obtained by training using the model training method provided in any optional embodiment of the present application.

[0221] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0222] In an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the above-mentioned model training method or image classification method. Compared with the related art, it can achieve the following: it can not only expand the number of samples by segmenting the original first sample image, but also ensure that the expanded second sample images contain real target objects, thereby ensuring the consistency of the actual content of the second sample image and its label category, so that the initial classification model can be effectively trained through the second sample image, thereby improving the accuracy of the trained classification model. In addition, the predicted area where the sample target object is located is used as the basis for dividing the first sample image, so that the second sample images obtained by the division are all related to the sample target object, thereby eliminating irrelevant background areas, reducing the interference of background areas in the image, facilitating the rapid convergence of the model, and improving the training speed of the model.

[0223] In an alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0224] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0225] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0226] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0227] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0228] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0229] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0230] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0231] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A model training method, characterized in that: include: acquiring a plurality of first sample images; Performing at least one training operation on the initial classification model based on the multiple first sample images, and using the initial classification model that meets the preset training end condition as the trained classification model; The training operation includes: Determining a predicted region where a sample target object is located in each first sample image; For each first sample image, segment the first sample image according to the determined prediction region to obtain at least one second sample image corresponding to the first sample image; the second sample image includes at least a portion of the prediction region where the sample target object is located; Determining the predicted categories corresponding to the at least one second sample image, and determining a loss function based on the true categories corresponding to the respective sample target objects and the predicted categories corresponding to the respective second sample images; Adjusting the parameters of the initial classification model based on the loss function, and using the initial classification model after adjusting the parameters as the initial classification model corresponding to the next training operation; The plurality of first sample images include a plurality of query images, a positive example image of the same category as at least one query image, and a negative example image of a different category from at least one query image. The determining of the loss function based on the true category corresponding to each sample target object and the predicted category corresponding to each second sample image includes: Determining the label category corresponding to each second sample image based on the true category corresponding to each sample target object; Determining a first loss function based on first differences between label categories corresponding to respective second sample images and their predicted categories; determining a second loss function based on a second difference between any two second sample images belonging to the same category and a third difference between any two second sample images not belonging to the same category; The loss function is determined based on the first loss function and the second loss function.

2. The model training method according to claim 1, characterized in that For each first sample image, segmenting the first sample image according to the determined prediction area to obtain at least one second sample image corresponding to the first sample image includes: determining at least one candidate region of the first sample image based on at least one detection box corresponding to the predicted region where the sample target object is located in the first sample image; Based on the determined at least one candidate area, at least one corresponding second sample image is generated respectively.

3. The model training method according to claim 2, characterized in that The candidate regions include a first candidate region and a second candidate region; The determining, based on at least one detection frame corresponding to the predicted region where the sample target object is located in the first sample image, at least one candidate region of the first sample image includes: Determine a first candidate region corresponding to each detection frame; Determine the minimum bounding rectangle area of ​​each detection frame to obtain the second candidate area.

4. The model training method according to claim 2, characterized in that Generating at least one corresponding second sample image based on the determined at least one candidate area includes: Based on the at least one candidate region, obtaining at least one third sample image corresponding to the at least one candidate region; Random data enhancement processing is performed on the at least one third sample image to obtain at least one second sample image.

5. The model training method according to claim 4, characterized in that The random data enhancement process includes at least one of the following: Random erase processing; Random cropping process; Random rotation processing; Random noise processing.

6. An image classification method, characterized in that: include: Acquire a first image to be classified; The trained classification model is used to perform classification operations to obtain the image category of the first image: The classification operation includes: Performing target detection on the first image to obtain a target area where a target object in the first image is located; Segmenting the first image based on a target region where a target object is located in the first image to obtain at least one second image corresponding to the first image; the second image includes at least a portion of the target region where the target object is located; classifying the at least one second image to obtain image categories corresponding to the respective second images, and determining the image category of the first image based on the image categories corresponding to the respective second images; Wherein, the classification model is obtained by training based on the model training method described in any one of claims 1 to 5.

7. A model training device, characterized in that: include: A first sample image acquisition module, configured to acquire a plurality of first sample images; a training module, configured to perform at least one training operation on the initial classification model based on the plurality of first sample images, and use the initial classification model that meets a preset training end condition as a trained classification model; The training operation includes: Determining a predicted region where a sample target object is located in each first sample image; For each first sample image, segment the first sample image according to the determined prediction region to obtain at least one second sample image corresponding to the first sample image; the second sample image includes at least a portion of the prediction region where the sample target object is located; Determining the predicted categories corresponding to the at least one second sample image, and determining a loss function based on the true categories corresponding to the respective sample target objects and the predicted categories corresponding to the respective second sample images; Adjusting the parameters of the initial classification model based on the loss function, and using the initial classification model after adjusting the parameters as the initial classification model corresponding to the next training operation; The plurality of first sample images include a plurality of query images, a positive example image of the same category as at least one query image, and a negative example image of a different category from at least one query image. The training module is used to determine the loss function based on the true category corresponding to each sample target object and the predicted category corresponding to each second sample image: Determining the label category corresponding to each second sample image based on the true category corresponding to each sample target object; Determining a first loss function based on first differences between label categories corresponding to respective second sample images and their predicted categories; determining a second loss function based on a second difference between any two second sample images belonging to the same category and a third difference between any two second sample images not belonging to the same category; The loss function is determined based on the first loss function and the second loss function.

8. An image classification device, characterized in that: include: A first image acquisition module, configured to acquire a first image to be classified; A classification module is configured to perform a classification operation using a trained classification model to obtain an image category of the first image: The classification operation includes: Performing target detection on the first image to obtain a target area where a target object in the first image is located; Segmenting the first image based on a target region where a target object is located in the first image to obtain at least one second image corresponding to the first image; the second image includes at least a portion of the target region where the target object is located; classifying the at least one second image to obtain image categories corresponding to the respective second images, and determining the image category of the first image based on the image categories corresponding to the respective second images; Wherein, the classification model is obtained by training based on the model training method described in any one of claims 1 to 5.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images