Image cropping and model training method, system, device and computer storage medium

Abstract

The application discloses an image cropping and model training method, system, device and computer storage medium, relates to the field of image processing, and obtains a target image to be cropped and a corresponding target label; determines class activation features of the target image; clusters the class activation features to obtain clustering results; crops the target image to obtain a cropped image; determines semantic correlation between the cropped image and the target image according to the clustering results; and determines a cropping label of the cropped image based on the semantic correlation and the target label. The application clusters the class activation features of the target image to obtain the clustering results reflecting feature types of the target image, then analyzes the similarity between the cropped image and the target image according to information of various features, and finally generates a label for the cropped image according to the similarity between the cropped image and the target image, so that the consistency between the cropped image and the cropping label is ensured, and the accuracy of image cropping is ensured.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to image cropping and model training methods, systems, devices, and computer storage media. Background Technology

[0002] Currently, in the process of image processing, data augmentation can be performed on the image, such as image cropping. Cropping means taking an image of any size from any position in the image. However, cropped images may contain incomplete or only a very small amount of feature information, resulting in inconsistencies between the image and the label attributes, which affects the accuracy of image cropping.

[0003] In conclusion, how to accurately crop images is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide an image cropping method that can, to some extent, solve the technical problem of how to accurately crop images. This invention also provides a model training method, an image cropping system, an apparatus, and a computer-readable storage medium.

[0005] According to a first aspect of the present invention, an image cropping method is provided, comprising:

[0006] Obtain the target image to be cropped and its corresponding target label;

[0007] Determine the class activation features of the target image;

[0008] Clustering is performed on the class activation features to obtain the clustering results;

[0009] The target image is cropped to obtain a cropped image;

[0010] Based on the clustering results, the semantic relevance between the cropped image and the target image is determined;

[0011] Based on the semantic relevance and the target label, the cropping label of the cropped image is determined.

[0012] In an exemplary embodiment, determining the class activation features of the target image includes:

[0013] Feature extraction is performed on the target image to obtain a feature map;

[0014] The feature map is subjected to class activation to obtain a class activation heatmap;

[0015] The activation heatmap is binarized to obtain a binary mask image;

[0016] Perform an AND operation between the feature map and the binary mask image to obtain the AND result image;

[0017] Based on the binary mask image and the AND operation result image, the class activation features are determined.

[0018] In an exemplary embodiment, performing class activation on the feature map to obtain a class activation heatmap includes:

[0019] The feature map is activated by class activation formula to obtain the class activation heatmap;

[0020] The class activation formula includes:

[0021]

[0022] in, This represents the class activation heatmap; feat c，k The feature map represents the k-th channel of the feature extraction output of the sample with category c in the target image; C represents the number of categories; The feature map of the k-th channel represents the weight of the class c. MN represents the gradient of the output probability value of category c with respect to the pixel (m, n) on the feature map of the k-th channel, where MN represents the total number of pixels on the feature map of the k-th channel.

[0023] In an exemplary embodiment, determining the class activation feature based on the binary mask image and the AND operation result image includes:

[0024] Determine the location of the pixel with the highest intensity value in the binary mask image;

[0025] In the image resulting from the AND operation, the feature data corresponding to the pixel position is determined as the class activation feature.

[0026] In an exemplary embodiment, the step of clustering the class activation features to obtain clustering results includes:

[0027] The clustering results are obtained by clustering the class activation features based on the K-means clustering algorithm.

[0028] In an exemplary embodiment, determining the semantic relevance between the cropped image and the target image based on the clustering result includes:

[0029] Based on the clustering results, determine the first number of pixels in the feature map that belong to the class-activated semantic region;

[0030] Based on the clustering results, a second number of pixels belonging to the class-activated semantic region in the cropped image is determined;

[0031] Using the bag-of-vocabularies model, a first semantic probability distribution of the feature map is determined based on the first quantity value;

[0032] The second semantic probability distribution of the cropped image is determined based on the second quantity value using the bag-of-vocabulary model.

[0033] The semantic relevance is determined based on the first quantity value, the second quantity value, the first semantic probability distribution, and the second semantic probability distribution.

[0034] In an exemplary embodiment, determining the first number of pixels belonging to the class-activated semantic region in the feature map based on the clustering result includes:

[0035] For each first pixel in the feature map, in the clustering result, the first cluster center closest to the first pixel is determined, and a first distance value between the first pixel and the first cluster center is determined. If the first distance value is less than or equal to the cluster radius of the first cluster center, then the first pixel is marked as a pixel belonging to the class-activated semantic region in the feature map.

[0036] The first count value of pixels belonging to the class-activated semantic region in the feature map is calculated.

[0037] In an exemplary embodiment, determining a second quantity of pixels belonging to a class-activated semantic region in the cropped image based on the clustering result includes:

[0038] For each second pixel in the cropped image, in the clustering result, the second cluster center closest to the second pixel is determined, and a second distance value between the second pixel and the second cluster center is determined. If the second distance value is less than or equal to the cluster radius of the second cluster center, then the second pixel is marked as a pixel belonging to the class-activated semantic region in the cropped image.

[0039] The second count value of pixels belonging to the class-activated semantic region in the cropped image is calculated.

[0040] In an exemplary embodiment, determining the first semantic probability distribution of the feature map based on the first quantity value using a bag-of-vocabulary model includes:

[0041] For each cluster center in the clustering results, the third quantity value of the first pixel in the class-activated semantic region of the cluster center is calculated, and the ratio of the third quantity value to the first quantity value is used as the first semantic probability value corresponding to the cluster center.

[0042] All the first semantic probability values ​​are combined to form a first probability distribution feature vector;

[0043] The first probability distribution feature vector is used as the first semantic probability distribution;

[0044] The step of determining the second semantic probability distribution of the cropped image based on the second quantity value using the bag-of-vocabulary model includes:

[0045] For each cluster center in the clustering results, a fourth quantity value of the second pixel in the class-activated semantic region of the cluster center is calculated, and the ratio of the fourth quantity value to the second quantity value is used as the second semantic probability value corresponding to the cluster center.

[0046] All the second semantic probability values ​​are combined to form a second probability distribution feature vector;

[0047] The feature vector of the second probability distribution is used as the second semantic probability distribution.

[0048] In an exemplary embodiment, determining the semantic relevance based on the first quantity value, the second quantity value, the first semantic probability distribution, and the second semantic probability distribution includes:

[0049] The semantic region size similarity is determined based on the ratio of the second quantity value to the first quantity value;

[0050] The semantic distribution similarity is determined based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution;

[0051] The semantic region size similarity and the semantic distribution similarity are used as the semantic relevance.

[0052] In an exemplary embodiment, determining the semantic region size similarity based on the ratio of the second quantity value to the first quantity value includes:

[0053] The semantic region size similarity is determined based on the ratio of the second quantity value to the first quantity value using the semantic region size similarity calculation formula.

[0054] The formula for calculating the similarity of semantic regions includes:

[0055]

[0056] in, The semantic region size similarity represents the similarity of the i-th sample of category C in the target image; This represents the second quantity value corresponding to the i-th sample of category C in the target image; β represents the first quantity value corresponding to the i-th sample of category c in the target image; β represents the hyperparameter.

[0057] Determining semantic distribution similarity based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution includes:

[0058] The semantic distribution similarity is determined based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution using the semantic distribution similarity calculation formula.

[0059] The semantic distribution similarity calculation formula includes:

[0060]

[0061] in, This represents the semantic distribution similarity corresponding to the i-th sample of category c in the target image; This represents the first semantic probability distribution corresponding to the i-th sample of category c in the target image; The second semantic probability distribution represents the i-th sample of category c in the target image; || represents taking the absolute value; γ represents the hyperparameter.

[0062] In one exemplary embodiment, the target label includes the label of the target image itself and the predicted label of the target image by the target neural network model performing image classification;

[0063] Determining the cropping label of the cropped image based on the semantic relevance and the target label includes:

[0064] The product of the semantic region size similarity and the tag is used as the pseudo-tag of the cropped image;

[0065] The product of the semantic distribution similarity and the predicted label is used as the soft label of the cropped label;

[0066] The pseudo-label and the soft label are used as the cropping label.

[0067] In an exemplary embodiment, before determining the cropping label of the cropped image based on the semantic relevance and the target label, the method further includes:

[0068] Determine whether the semantic relevance is greater than a preset threshold;

[0069] If the semantic relevance is greater than the preset threshold, the cropped image is determined to be valid, and the step of determining the cropping label of the cropped image based on the semantic relevance and the target label is executed.

[0070] If the semantic relevance is less than or equal to the preset threshold, the cropped image is determined to be invalid, and the process returns to cropping the target image to obtain the cropped image and subsequent steps.

[0071] In one exemplary embodiment, obtaining the target image to be cropped includes:

[0072] Obtain the original image to be cropped;

[0073] The original image is preprocessed to obtain the target image;

[0074] The preprocessing includes brightness normalization, noise filtering, scale normalization, and flipping.

[0075] According to a second aspect of the present invention, a model training method is provided, including the steps of any of the image cropping methods described above;

[0076] Based on the target image, the target label, the cropped image, and the cropped label, a target neural network model for image classification is trained.

[0077] In one exemplary embodiment, the cropped label includes a pseudo label and a soft label;

[0078] The training process of the target neural network model includes:

[0079] The loss value is calculated based on the soft label and the pseudo label using the loss function calculation formula.

[0080] The loss function calculation formula includes:

[0081] L = αL1 + (1-α)L2;

[0082]

[0083]

[0084] Where L represents the calculated loss value; α represents the hyperparameter; and C represents the total number of categories; This represents the predicted probability value of the target neural network model for the j-th category of the cropped image corresponding to the i-th sample of category c in the target image; The pseudo-label value represents the value of the cropped image corresponding to the i-th sample of category c in the target image, which belongs to the j-th category. The value of the soft label represents the value of the cropped image corresponding to the i-th sample of category c in the target image, belonging to the j-th category.

[0085] In an exemplary embodiment, feature extraction is performed on the target image to obtain a feature map, and class activation is performed on the feature map to obtain a class activation heatmap, including:

[0086] The target neural network model is initially trained based on the target image and the labeled data to obtain the initially trained target neural network model.

[0087] The target image is input into the pre-trained target neural network model to obtain the class activation heatmap output by the pre-trained target neural network model.

[0088] In an exemplary embodiment, after training the target neural network model for image classification based on the target image, the target label, the cropped image, and the cropped label, the method further includes:

[0089] Obtain the image to be classified;

[0090] Transmit the image to be classified to the trained target neural network model;

[0091] Obtain the classification result of the image to be classified from the output of the trained target neural network model.

[0092] According to a third aspect of the present invention, an image cropping system is provided, comprising:

[0093] The image acquisition module is used to acquire the target image to be cropped and its corresponding target label;

[0094] A feature determination module is used to determine the class activation features of the target image;

[0095] The feature clustering module is used to cluster the class activation features to obtain clustering results;

[0096] The image cropping module is used to crop the target image to obtain a cropped image;

[0097] A semantic determination module is used to determine the semantic relevance between the cropped image and the target image based on the clustering results;

[0098] The label determination module is used to determine the cropping label of the cropped image based on the semantic relevance and the target label.

[0099] According to a fourth aspect of the present invention, an image cropping device is provided, comprising:

[0100] Memory, used to store computer programs;

[0101] A processor for implementing the steps of any of the image cropping methods described above when executing the computer program.

[0102] According to a fifth aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the image cropping methods described above.

[0103] This invention provides an image cropping method that involves: acquiring a target image to be cropped and its corresponding target label; determining the class activation features of the target image; clustering the class activation features to obtain clustering results; cropping the target image to obtain a cropped image; determining the semantic relevance between the cropped image and the target image based on the clustering results; and determining the cropping label of the cropped image based on the semantic relevance and the target label. The beneficial effects of this invention are: by clustering the class activation features of the target image, clustering results reflecting the feature types of the target image are obtained; then, the semantic relevance between the cropped image and the target image is determined based on the clustering results, which is equivalent to analyzing the similarity between the cropped image and the target image based on information from various features; finally, the cropping label of the cropped image is determined based on the semantic relevance and the target label, which is equivalent to generating a label for the cropped image based on the similarity between the cropped image and the target image, ensuring the consistency between the cropped image and the cropping label, thereby ensuring the accuracy of image cropping. This invention also provides a model training method, image cropping system, device, and computer-readable storage medium that solve corresponding technical problems. Attached Figure Description

[0104] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0105] Figure 1 This is a first schematic diagram of an image cropping method provided in an embodiment of the present invention;

[0106] Figure 2 This is a second schematic diagram of an image cropping method provided in an embodiment of the present invention;

[0107] Figure 3 This is a third schematic diagram of an image cropping method provided in an embodiment of the present invention;

[0108] Figure 4 A schematic diagram of clustering based on class activation features;

[0109] Figure 5 A schematic diagram for calculating the semantic relevance of a cropped image;

[0110] Figure 6 This is a schematic diagram illustrating the training of the target neural network model.

[0111] Figure 7 This is a schematic diagram of the structure of an image cropping system provided in an embodiment of the present invention;

[0112] Figure 8 This is a schematic diagram of the structure of an image cropping device provided in an embodiment of the present invention;

[0113] Figure 9 This is another structural schematic diagram of an image cropping device provided in an embodiment of the present invention. Detailed Implementation

[0114] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0115] Please see Figure 1 , Figure 1 This is a first schematic diagram of an image cropping method provided in an embodiment of the present invention.

[0116] An image cropping method provided in this embodiment of the invention may include the following steps:

[0117] Step S101: Obtain the target image to be cropped and its corresponding target label.

[0118] In practical applications, the target image to be cropped and its target label can be obtained first. The information of the target image and its corresponding target label can be determined according to actual needs. For example, the target image can be a pedestrian image, a server performance snapshot image, a surveillance image, etc. This invention does not make any specific limitations here.

[0119] In specific application scenarios, in order to ensure that the target image meets the processing requirements, the original image to be cropped can be obtained first; the original image can be preprocessed to obtain the target image; the preprocessing can include brightness normalization, noise filtering, scale normalization, flipping, interpolation, etc.

[0120] Understandably, the specific preprocessing method can be determined based on the application scenario. For example, during the flipping process, horizontal or vertical flipping can be performed; during the interpolation process, the VGG16 model can be used to adjust the image size to (224, 224) to adjust the image size to the required size; during the brightness normalization process, the image can be normalized based on the mean and standard deviation of the image brightness. For example, the brightness normalization operation can be performed according to the mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] of the brightness normalization datasets such as ImageNet and VOC (Visual Object Classes).

[0121] Step S102: Determine the class activation features of the target image.

[0122] In practical applications, after obtaining the target image to be cropped and its corresponding target label, the class activation features of the target image can be determined. These class activation features can be used to characterize the classification contribution heatmap of the target image, which is convenient for subsequent application of these class activation features to generate cropping labels for the cropped image.

[0123] Step S103: Cluster the class activation features to obtain the clustering results.

[0124] In practical applications, after determining the class activation features of the target image, the class activation features can be clustered to obtain the corresponding clustering results, which can then be used to reflect the corresponding information of various features in the target image.

[0125] Step S104: Crop the target image to obtain a cropped image.

[0126] Step S105: Determine the semantic correlation between the cropped image and the target image based on the clustering results.

[0127] Step S106: Determine the cropping label of the cropped image based on semantic relevance and target label.

[0128] In practical applications, after clustering the class activation features and obtaining the clustering results, the target image can be cropped to obtain the cropped image. Based on the clustering results, the semantic relevance between the cropped image and the target image is determined. Based on the semantic relevance and the target label, the cropping label of the cropped image is determined.

[0129] In specific application scenarios, during the process of cropping a target image to obtain a cropped image, the cropping region can be randomly generated. For example, if the cropping position is set to (cx, cy), the cropping region ratio is ratio, and the size of the target image is (sw, sh), then the cropping region will be (cx, cy, ratio*sw, ratio*sh), and the cropped image will be the image corresponding to the cropped region in the target image. It should be noted that, in order to prevent false positives in the cropped image, the cropping ratio can be set, such as requiring the cropping ratio to be greater than a cropping threshold, such as being greater than 0.8, etc. This invention does not impose specific limitations on this.

[0130] This invention provides an image cropping method that involves: acquiring a target image to be cropped and its corresponding target label; determining the class activation features of the target image; clustering the class activation features to obtain clustering results; cropping the target image to obtain a cropped image; determining the semantic relevance between the cropped image and the target image based on the clustering results; and determining the cropping label of the cropped image based on the semantic relevance and the target label. The beneficial effects of this invention are: by clustering the class activation features of the target image, clustering results reflecting the feature types of the target image are obtained; then, the semantic relevance between the cropped image and the target image is determined based on the clustering results, which is equivalent to analyzing the similarity between the cropped image and the target image based on information from various features; finally, the cropping label of the cropped image is determined based on the semantic relevance and the target label, which is equivalent to generating a label for the cropped image based on the similarity between the cropped image and the target image, ensuring the consistency between the cropped image and the cropping label, thereby ensuring the accuracy of image cropping.

[0131] Please see Figure 2 , Figure 2 This is a second schematic diagram of an image cropping method provided in an embodiment of the present invention.

[0132] Based on the above embodiments, the image cropping method provided by this invention may include the following steps:

[0133] Step S201: Obtain the target image to be cropped and its corresponding target label.

[0134] Step S202: Extract features from the target image to obtain a feature map.

[0135] In practical applications, when determining the class activation features of a target image, feature extraction can be performed on the target image first to obtain a feature map that reflects each feature of the target image.

[0136] Step S203: Perform class activation on the feature map to obtain a class activation heatmap.

[0137] In practical applications, after obtaining the feature map, class activation can be performed on the feature map to obtain a class activation heatmap.

[0138] It should be noted that Class Activation Mapping (CAM) is a visual and interpretable method for neural network models, outputting a heatmap of the classification contribution of an image. Generally, the larger the pixel value of the heatmap, the greater its impact on the output. In specific applications, class activation visualization methods include CAM, grad-CAM, and grad-CAM++. GradCAM and grad-CAM++ are applicable to any type of neural network model and do not require modification of the model's hierarchical structure; they can be obtained through backpropagation calculations. In this embodiment, the grad-CAM class activation method can be used to obtain the class activation heatmap. That is, during the process of class activation of the feature map to obtain the class activation heatmap, the class activation formula can be used to perform class activation on the feature map to obtain the class activation heatmap.

[0139] Class activation formulas include:

[0140]

[0141] in, Represents a class activation heatmap; feat c，k This represents the feature map of the k-th channel of the feature extraction output of a sample of category c in the target image; C represents the number of categories. This represents the weight of the feature map of the k-th channel with respect to category c; MN represents the gradient of the output probability value of category c with respect to the pixel (m, n) on the feature map of the k-th channel, where MN represents the total number of pixels on the feature map of the k-th channel.

[0142] Step S204: Binarize the class activation heatmap to obtain a binary mask image.

[0143] In practical applications, after performing class activation on the feature map to obtain the class activation heatmap, in order to determine the effective feature information in the class activation heatmap, the class activation heatmap can be binarized to obtain a binary mask image.

[0144] Understandably, the method for binarizing the class activation heatmap can be determined based on the application scenario. For example, a threshold segmentation method can be used to binarize the class activation heatmap to obtain a binary mask image of the feature map from 0 to 255.

[0145] Step S205: Perform an AND operation on the feature map and the binary mask image to obtain the result image.

[0146] Step S206: Determine class activation features based on the binary mask image and the image resulting from the AND operation.

[0147] In practical applications, after binarizing the class activation heatmap to obtain a binary mask image, the feature map and the binary mask image can be ANDed to obtain the AND result image. Then, based on the binary mask image and the AND result image, the class activation features can be determined.

[0148] In specific application scenarios, when determining class activation features based on a binary mask image and the result image of a bitwise AND operation, the location of the pixel with the highest intensity value in the binary mask image can be identified. In the result image of the bitwise AND operation, the feature data corresponding to that pixel location is determined as the class activation feature. For ease of understanding, assuming the pixel value in the binary mask image is between 0 and 255, the feature data in the result image of the bitwise AND operation corresponding to the pixel location with an intensity value of 255 in the binary mask image can be determined as the corresponding class activation feature.

[0149] Step S207: Cluster the class activation features to obtain the clustering results.

[0150] Step S208: Crop the target image to obtain a cropped image.

[0151] Step S209: Determine the semantic correlation between the cropped image and the target image based on the clustering results.

[0152] Step S210: Determine the cropping label of the cropped image based on semantic relevance and target label.

[0153] Please see Figure 3 , Figure 3 This is a third schematic diagram of an image cropping method provided in an embodiment of the present invention.

[0154] Based on the above embodiments, the image cropping method provided by this invention may include the following steps:

[0155] Step S301: Obtain the target image to be cropped and its corresponding target label.

[0156] Step S302: Determine the class activation features of the target image.

[0157] Step S303: Cluster the class activation features to obtain the clustering results.

[0158] In practical applications, the method for clustering class activation features can be determined based on the application scenario. For example, the K-means clustering algorithm can be used to cluster class activation features to obtain clustering results. This process can be as follows: Figure 4 As shown.

[0159] Step S304: Crop the target image to obtain a cropped image.

[0160] Step S305: Based on the clustering results, determine the first number of pixels in the feature map that belong to the class activation semantic region.

[0161] Step S306: Based on the clustering results, determine the second quantity value of pixels in the cropped image that belong to the class-activated semantic region.

[0162] In practical applications, when determining the semantic relevance between the cropped image and the target image based on the clustering results, a first quantity of pixels belonging to the class-activated semantic region in the feature map can be determined based on the clustering results; a second quantity of pixels belonging to the class-activated semantic region in the cropped image can be determined based on the clustering results; and then the first and second quantities can be applied to determine the semantic relevance.

[0163] In specific application scenarios, considering that in addition to class-related semantic regions, there are also class-independent regions on the image, such as regions containing other categories or background information, class-independent regions can be determined by the distance from the feature vector to the cluster center in order to remove class-independent regions. That is, in the process of determining the first number of pixels belonging to the class-activated semantic region in the feature map based on the clustering results, for each first pixel in the feature map, the nearest first cluster center can be determined in the clustering results, and the first distance value between the first pixel and the first cluster center can be determined. This distance value is used to describe the similarity between features. If the first distance value is less than or equal to the cluster radius of the first cluster center, the first pixel is marked as a pixel belonging to the class-activated semantic region in the feature map; the first number of pixels belonging to the class-activated semantic region in the feature map is counted. Accordingly, in the process of determining the second number of pixels belonging to the class-activated semantic region in the cropped image based on the clustering results, for each second pixel in the cropped image, the second cluster center closest to the second pixel can be determined from the clustering results, and the second distance value between the second pixel and the second cluster center can be determined. If the second distance value is less than or equal to the cluster radius of the second cluster center, then the second pixel is marked as a pixel belonging to the class-activated semantic region in the cropped image; and the second number of pixels belonging to the class-activated semantic region in the cropped image is counted.

[0164] In specific application scenarios, clustering models can be used to cluster class activation features. Let D be the set of obtained class activation feature vectors. c The dimension is [P, Ch], where P represents the number of class-related feature vectors for category c. Let D... cThe input is fed into a clustering model for training, dividing the sample set into K clusters such that the distance from each sample to the center of its respective cluster is minimized. The output of the clustering training completes includes K cluster centers and cluster radii. The feature vector of each cluster center represents a semantic feature. The set of cluster centers is denoted as `centerc`, with dimensions [K, Ch], and the cluster radius is denoted as `R`. c The feature map of the target image has a dimension of [K]. The data is input into a clustering model for clustering inference, and the output distance is... The feature vectors of the nearest cluster centers are the feature vectors of the pixel points (rr, cc) on the feature map. For example, the clustering reasoning process is as follows: Where, kmeans_model represents the clustering model, The value of can be {0, 1, 2, ..., K-1}; correspondingly, if Then it can be determined that the pixel (rr, cc) belongs to the class activation semantic region.

[0165] Step S307: Determine the first semantic probability distribution of the feature map based on the first quantity value using the bag-of-vocabulary model.

[0166] Step S308: Determine the second semantic probability distribution of the cropped image based on the second quantity value using the bag-of-vocabulary model.

[0167] Step S309: Determine semantic relevance based on the first quantity value, the second quantity value, the first semantic probability distribution, and the second semantic probability distribution.

[0168] In practical applications, after obtaining the first and second quantity values, the first semantic probability distribution of the feature map can be determined based on the first quantity value using the bag-of-vocabulary model; the second semantic probability distribution of the cropped image can be determined based on the second quantity value using the bag-of-vocabulary model; and semantic relevance can be determined based on the first quantity value, the second quantity value, the first semantic probability distribution, and the second semantic probability distribution.

[0169] In practical applications, when determining the first semantic probability distribution of the feature map based on a first quantity value using the bag-of-vocabulary model, for each cluster center in the clustering results, a third quantity value can be calculated for the first pixel in the class-activated semantic region of the cluster center. The ratio of the third quantity value to the first quantity value is used as the first semantic probability value corresponding to the cluster center. All the first semantic probability values ​​are then combined to form a first probability distribution feature vector, which is used as the first semantic probability distribution. Similarly, when determining the second semantic probability distribution of the cropped image based on a second quantity value using the bag-of-vocabulary model, for each cluster center in the clustering results, a fourth quantity value is calculated for the second pixel in the class-activated semantic region of the cluster center. The ratio of the fourth quantity value to the second quantity value is used as the second semantic probability value corresponding to the cluster center. All the second semantic probability values ​​are then combined to form a second probability distribution feature vector, which is used as the second semantic probability distribution.

[0170] In practical applications, semantic region size similarity and semantic distribution similarity can be used to measure the semantic relevance between the original image and the cropped image. The values ​​range from (0, 1). That is, in determining semantic relevance based on a first quantitative value, a second quantitative value, a first semantic probability distribution, and a second semantic probability distribution, semantic region size similarity can be determined based on the ratio of the second quantitative value to the first quantitative value; semantic distribution similarity can be determined based on the cosine distance between the first and second semantic probability distributions. Using semantic region size similarity and semantic distribution similarity as semantic relevance, such as... Figure 5 As stated above.

[0171] In specific application scenarios, in the process of determining the semantic region size similarity based on the ratio of the second quantity value to the first quantity value, the semantic region size similarity can be determined based on the ratio of the second quantity value to the first quantity value through the semantic region size similarity calculation formula.

[0172] The formulas for calculating semantic region size similarity include:

[0173]

[0174] in, This represents the semantic region size similarity of the i-th sample of category c in the target image; This represents the second quantity value corresponding to the i-th sample of category c in the target image; β represents the first quantity value corresponding to the i-th sample of category c in the target image; β represents the hyperparameter used to adjust the ratio of semantic region size, and thus adjust the degree of influence of semantic region size on the label. The value range of β is (0, 1], and the closer the value of β is to 1, the greater the influence of semantic region size on the label.

[0175] Correspondingly, in the process of determining the semantic distribution similarity based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution, the semantic distribution similarity can be determined based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution using the semantic distribution similarity calculation formula.

[0176] The formulas for semantic distribution similarity calculation include:

[0177]

[0178] in, This represents the semantic distribution similarity of the i-th sample of category c in the target image; Let represent the first semantic probability distribution corresponding to the i-th sample of category c in the target image; Let represent the second semantic probability distribution corresponding to the i-th sample of category c in the target image; || represents taking the absolute value; γ represents a hyperparameter used to adjust the semantic distribution similarity, and thus adjust the degree of influence of the semantic distribution on the label. The value range of γ is (0, 1]. The closer the value of γ is to 1, the greater the influence of the semantic distribution on the label.

[0179] Step S310: Determine the cropping label of the cropped image based on semantic relevance and target label.

[0180] In practical applications, during the process of determining the cropping label of a cropped image based on semantic relevance and target label, the target label can include the target image's own label and the predicted label of the target image by the target neural network model performing image classification. Accordingly, the product of semantic region size similarity and label can be used as the pseudo label of the cropped image; the product of semantic distribution similarity and predicted label can be used as the soft label of the cropping label; and the pseudo label and soft label can be used as the cropping label.

[0181] It's important to note that pseudo-labels describe the value of a label that closely approximates the cropped image. Therefore, there is a correlation between the pseudo-label of the cropped image and the label of the original image. This correlation can be described using semantic region size similarity. Furthermore, considering that the cropped image has fewer feature information than the original image, the corresponding label value is also smaller. Therefore, the pseudo-label is the target image's label multiplied by the semantic region size ratio. Soft labels describe the value of a predicted label that closely approximates the cropped image. Again, there is a correlation between the soft label of the cropped image and the predicted label of the original image. Considering the change in feature distribution between the cropped image and the original image, semantic distribution similarity is used to describe the correlation. Therefore, the soft label is the target image's predicted label multiplied by the semantic distribution similarity.

[0182] In practical applications, image cropping can be used in scenarios such as image recognition and image classification. Considering that deep neural networks are widely used in increasingly complex scenarios due to their powerful data learning capabilities, high-performance neural network models often have data-driven characteristics, requiring diverse and massive amounts of data for training. Therefore, the dataset is usually preprocessed before model training. Besides reducing dataset noise, the main purpose of preprocessing is data augmentation, i.e., increasing the quantity and diversity of the dataset. However, the randomness of the data augmentation process also introduces some noise into the dataset. Taking cropping as an example, cropping an image of arbitrary size at any position may result in incomplete or minimal feature information, leading to inconsistencies between the sample image and the label attributes, and even false positives, thus affecting model performance. Therefore, to ensure the training accuracy of the neural network model, this invention, after determining the cropping label of the cropped image based on semantic relevance and the target label, can also train a target neural network model for image classification based on the target image, target label, cropped image, and cropping label. The type of target neural network model can be determined according to the application scenario, and this application does not specifically limit it.

[0183] In specific application scenarios, the loss function is the objective function optimized in a neural network, used to measure the deviation between the model's output prediction value y and the training target value Y. The process of training a neural network is the process of minimizing the loss function. The smaller the loss function, the closer the model's prediction value is to the true value, and the better the model's accuracy. Commonly used loss functions include mean squared error loss, cross-entropy loss, binary cross-entropy loss, and KL divergence (Kullback-Leibler divergence, also known as relative entropy). In this embodiment, binary cross-entropy is used to calculate the deviation between the model's output value and the pseudo-label, and KL divergence is used to calculate the deviation between the model's output value and the soft label. That is, during the training process of the target neural network model, the loss value can be calculated based on the soft label and pseudo-label using the loss function calculation formula.

[0184] The formula for calculating the loss function includes:

[0185] L = αL1 + (1-α)L2:

[0186]

[0187]

[0188] Where L represents the calculated loss value; α represents the hyperparameter; and C represents the total number of categories; This represents the predicted probability value of the target neural network model for the i-th sample of class c in the target image, corresponding to the j-th class of the cropped image; This represents the value of the pseudo-label of the cropped image corresponding to the i-th sample of category c in the target image, which belongs to the j-th category. This represents the value of the soft label for the j-th category of the cropped image corresponding to the i-th sample of category c in the target image.

[0189] In specific application scenarios, a target neural network model can be used to extract features from a target image to obtain a feature map. In other words, the target neural network model can be initially trained based on the target image and target label to obtain a pre-trained target neural network model. The target image is then input into the pre-trained target neural network model to obtain the feature map output by the convolutional hidden layer of the pre-trained target neural network model.

[0190] In specific application scenarios, after training a target neural network model for image classification based on the target image, target label, cropped image, and cropped label, the image to be classified can be obtained; the image to be classified can be transmitted to the trained target neural network model; and the classification result of the image to be classified output by the trained target neural network model can be obtained.

[0191] In specific application scenarios, to further prevent false positives in cropped images, semantic relevance can be set to avoid this problem. That is, before determining the cropping label of the cropped image based on semantic relevance and target label, it can be determined whether the semantic relevance is greater than a preset threshold. If the semantic relevance is greater than the preset threshold, the cropped image is determined to be valid, and the step of determining the cropping label of the cropped image based on semantic relevance and target label is executed. If the semantic relevance is less than or equal to the preset threshold, the cropped image is determined to be invalid, and the process of cropping the target image is returned to obtain the cropped image and subsequent steps.

[0192] To facilitate understanding of the image cropping method provided by this invention, its application will now be explained in conjunction with the training process of a target neural network model for image classification. This process can be described as follows: Figure 6 As shown, the convolutional neural network model structure typically includes a convolutional hidden layer structure and a fully connected layer structure. The former is used for feature extraction, obtaining feature images from several channels, while the latter serves as a classifier model, outputting classification prediction values ​​to achieve classification. This can include the following steps:

[0193] Obtain the original sample images and corresponding labeled sample labels for training;

[0194] The original sample image is preprocessed to obtain the target sample image;

[0195] The target sample image and labeled sample labels are used to perform forward propagation calculations on the target neural network model, outputting a class probability vector. In this process, let the input image be denoted as... Let represent the i-th sample of category c, and let be the predicted category probability vector output by the target neural network model. The probability vector has dimension C, representing the number of classes, and f represents the function computed in the forward pass of the model. Forward computation can be viewed as a combination of feature extraction and a classifier model. The feature map output by feature extraction is denoted as... The dimensions are denoted as [M, N, Ch], where M and N represent the width and height of the feature extraction output image, and Ch represents the number of feature extraction output images;

[0196] Based on the predicted label values ​​and labeled sample labels output by the target neural network model, backpropagation is performed on the target neural network model to update the gradient image, model weights, and bias parameters. During this process, binary cross-entropy can be used to calculate the classification loss of the original image, and the deviation between the predicted label values ​​output by the model and the labeled sample labels can be calculated using the formula... To calculate the loss value during this initial training process, where L3 represents the corresponding loss value. This represents the j-th predicted label of the i-th sample feature of category c in the target sample image by the target neural network model; This represents the label of the j-th labeled sample in the i-th sample feature of category c in the target sample image;

[0197] Obtain the predicted sample labels of the target sample image by the initially trained target neural network model;

[0198] The feature map is obtained by extracting features from the convolutional hidden layers of the target neural network model after initial training and performing feature extraction on the target sample image.

[0199] Class activation is applied to the feature map to obtain a class activation heatmap;

[0200] The class activation heatmap is binarized to obtain a binary mask image;

[0201] Perform an AND operation between the feature map and the binary mask image to obtain the result image.

[0202] Determine the location of the pixel with the highest intensity value in the binary mask image; in the result image of the AND operation, determine the feature data corresponding to the pixel location as the class activation feature;

[0203] Clustering of class activation features is performed based on the K-means clustering algorithm to obtain clustering results;

[0204] The target sample image is cropped to obtain the cropped image;

[0205] For each first pixel in the feature map, in the clustering results, determine the first cluster center closest to the first pixel, determine the first distance value between the first pixel and the first cluster center, if the first distance value is less than or equal to the cluster radius of the first cluster center, then mark the first pixel as a pixel belonging to the class-activated semantic region in the feature map; count the first number of pixels belonging to the class-activated semantic region in the feature map.

[0206] For each second pixel in the cropped image, in the clustering results, determine the second cluster center closest to the second pixel, determine the second distance value between the second pixel and the second cluster center, if the second distance value is less than or equal to the cluster radius of the second cluster center, then mark the second pixel as a pixel belonging to the class-activated semantic region in the cropped image; count the second number of pixels belonging to the class-activated semantic region in the cropped image.

[0207] For each cluster center in the clustering results, the third quantity value of the first pixel in the class activation semantic region of the cluster center is calculated, and the ratio of the third quantity value to the first quantity value is used as the first semantic probability value corresponding to the cluster center; all the first semantic probability values ​​are combined to form a first probability distribution feature vector; the first probability distribution feature vector is used as the first semantic probability distribution;

[0208] For each cluster center in the clustering results, the fourth quantity value of the second pixel in the class activation semantic region of the cluster center is calculated, and the ratio of the fourth quantity value to the second quantity value is used as the second semantic probability value corresponding to the cluster center; all the second semantic probability values ​​are combined to form a second probability distribution feature vector; the second probability distribution feature vector is used as the second semantic probability distribution.

[0209] The semantic region size similarity is determined based on the ratio of the second quantity value to the first quantity value;

[0210] Semantic distribution similarity is determined based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution;

[0211] The product of semantic region size similarity and labeled sample label is used as the pseudo label of the cropped image;

[0212] The product of semantic distribution similarity and predicted sample label is used as the soft label for cropping the label;

[0213] Based on the target sample image, labeled sample, cropped image, pseudo label and soft label, the target neural network model is trained again to obtain the trained target neural network model;

[0214] Obtain the image to be classified;

[0215] Transmit the image to be classified to the trained target neural network model;

[0216] Obtain the classification results of the image to be classified from the output of the trained target neural network model.

[0217] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an image cropping system provided in an embodiment of the present invention.

[0218] An image cropping system provided in this embodiment of the invention may include:

[0219] Image acquisition module 101 is used to acquire the target image to be cropped and the corresponding target label;

[0220] Feature determination module 102 is used to determine the class activation features of the target image;

[0221] The feature clustering module 103 is used to cluster class activation features to obtain clustering results;

[0222] Image cropping module 104 is used to crop the target image to obtain a cropped image;

[0223] The semantic determination module 105 is used to determine the semantic relevance between the cropped image and the target image based on the clustering results.

[0224] The label determination module 106 is used to determine the cropping label of the cropped image based on semantic relevance and target label.

[0225] The image cropping system provided in this embodiment of the invention includes a feature determination module that can be specifically used for: extracting features from a target image to obtain a feature map; performing class activation on the feature map to obtain a class activation heatmap; binarizing the class activation heatmap to obtain a binary mask image; performing an AND operation on the feature map and the binary mask image to obtain an AND result image; and determining class activation features based on the binary mask image and the AND result image.

[0226] The image cropping system provided in this embodiment of the invention includes a feature determination module that can be specifically used to perform class activation on a feature map using a class activation formula to obtain a class activation heatmap.

[0227] Class activation formulas include:

[0228]

[0229] in, Represents a class activation heatmap; feat c，k This represents the feature map of the k-th channel of the feature extraction output of a sample of category c in the target image; C represents the number of categories. This represents the weight of the feature map of the k-th channel with respect to category c; MN represents the gradient of the output probability value of category c with respect to the pixel (m, n) on the feature map of the k-th channel, where MN represents the total number of pixels on the feature map of the k-th channel.

[0230] The image cropping system provided in this embodiment of the invention includes a feature determination module that can be specifically used to: determine the position of the pixel with the highest intensity value in a binary mask image; and determine the feature data corresponding to the pixel position as a class activation feature in the result image of the AND operation.

[0231] The image cropping system provided in this embodiment of the invention includes a feature clustering module that can be specifically used to: cluster class activation features based on the K-means clustering algorithm to obtain clustering results.

[0232] This invention provides an image cropping system in which a semantic determination module can be specifically used to: determine a first quantity of pixels belonging to a class-activated semantic region in a feature map based on clustering results; determine a second quantity of pixels belonging to a class-activated semantic region in the cropped image based on clustering results; determine a first semantic probability distribution of the feature map based on the first quantity using a bag-of-vocabulary model; determine a second semantic probability distribution of the cropped image based on the second quantity using a bag-of-vocabulary model; and determine semantic relevance based on the first quantity, the second quantity, the first semantic probability distribution, and the second semantic probability distribution.

[0233] The image cropping system provided in this embodiment of the invention includes a semantic determination module that can be specifically used to: for each first pixel in the feature map, determine the first cluster center closest to the first pixel in the clustering results, determine the first distance value between the first pixel and the first cluster center, and if the first distance value is less than or equal to the cluster radius of the first cluster center, then mark the first pixel as a pixel belonging to the class-activated semantic region in the feature map; and count the first number of pixels belonging to the class-activated semantic region in the feature map.

[0234] The image cropping system provided in this embodiment of the invention includes a semantic determination module that can be specifically used to: for each second pixel in the cropped image, determine the second cluster center closest to the second pixel in the clustering results, determine the second distance value between the second pixel and the second cluster center, and if the second distance value is less than or equal to the cluster radius of the second cluster center, then mark the second pixel as a pixel belonging to the class-activated semantic region in the cropped image; and count the second number of pixels belonging to the class-activated semantic region in the cropped image.

[0235] This invention provides an image cropping system in which a semantic determination module can be specifically used to: for each cluster center in the clustering results, calculate the third quantity value of the first pixel in the class-activated semantic region of the cluster center, and use the ratio of the third quantity value to the first quantity value as the first semantic probability value corresponding to the cluster center; form a first probability distribution feature vector from all the first semantic probability values; and use the first probability distribution feature vector as the first semantic probability distribution; for each cluster center in the clustering results, calculate the fourth quantity value of the second pixel in the class-activated semantic region of the cluster center, and use the ratio of the fourth quantity value to the second quantity value as the second semantic probability value corresponding to the cluster center; form a second probability distribution feature vector from all the second semantic probability values; and use the second probability distribution feature vector as the second semantic probability distribution.

[0236] The image cropping system provided in this embodiment of the invention includes a semantic determination module that can be specifically used to: determine the semantic region size similarity based on the ratio of a second quantity value to a first quantity value; determine the semantic distribution similarity based on the cosine distance between a first semantic probability distribution and a second semantic probability distribution; and use the semantic region size similarity and semantic distribution similarity as semantic relevance.

[0237] The image cropping system provided in this embodiment of the invention includes a semantic determination module that can be specifically used for:

[0238] The semantic region size similarity is determined based on the ratio of the second quantity value to the first quantity value using the semantic region size similarity calculation formula.

[0239] The formulas for calculating semantic region size similarity include:

[0240]

[0241] in, This represents the semantic region size similarity of the i-th sample of category c in the target image; This represents the second quantity value corresponding to the i-th sample of category c in the target image; The first count value represents the i-th sample of category c in the target image; β represents the hyperparameter.

[0242] The semantic distribution similarity is determined based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution using the semantic distribution similarity calculation formula.

[0243] The formulas for semantic distribution similarity calculation include:

[0244]

[0245] in, This represents the semantic distribution similarity of the i-th sample of category c in the target image; Let represent the first semantic probability distribution corresponding to the i-th sample of category c in the target image; Let || represent the second semantic probability distribution corresponding to the i-th sample of category c in the target image; || represents taking the absolute value; γ represents the hyperparameter.

[0246] This invention provides an image cropping system where the target label includes the target image's own label and the predicted label of the target image by the target neural network model performing image classification. The label determination module can be specifically used to: use the product of semantic region size similarity and the label as the pseudo label of the cropped image; use the product of semantic distribution similarity and the predicted label as the soft label of the cropping label; and use the pseudo label and the soft label as the cropping label.

[0247] An image cropping system provided in this embodiment of the invention may further include:

[0248] The judgment module is used to determine whether the semantic relevance is greater than a preset threshold before the label determination module determines the cropping label of the cropped image based on semantic relevance and the target label. If the semantic relevance is greater than the preset threshold, the cropped image is determined to be valid, and the label determination module is instructed to perform the step of determining the cropping label of the cropped image based on semantic relevance and the target label. If the semantic relevance is less than or equal to the preset threshold, the cropped image is determined to be invalid, and the image cropping module is instructed to return to perform cropping on the target image to obtain the cropped image and subsequent steps.

[0249] The image cropping system provided in this embodiment of the invention includes an image acquisition module that can be specifically used to: acquire the original image to be cropped; preprocess the original image to obtain the target image; wherein the preprocessing includes brightness normalization, noise filtering, scale normalization, and flipping.

[0250] An embodiment of the present invention provides a model training system that may include:

[0251] Image cropping system as described in any of the above embodiments;

[0252] The training module is used to train a target neural network model for image classification based on the target image, target label, cropped image, and cropped label.

[0253] This invention provides a model training system where the cropped labels include pseudo-labels and soft labels; the training process of the target neural network model may include:

[0254] The loss value calculation module is used to calculate the loss value based on soft labels and pseudo labels using the loss function calculation formula.

[0255] The formula for calculating the loss function includes:

[0256] L = αL1 + (1-α)L2;

[0257]

[0258]

[0259] Where L represents the calculated loss value; α represents the hyperparameter; and C represents the total number of categories; This represents the predicted probability value of the target neural network model for the i-th sample of class c in the target image, corresponding to the j-th class of the cropped image; This represents the value of the pseudo-label of the cropped image corresponding to the i-th sample of category c in the target image, which belongs to the j-th category. This represents the value of the soft label for the j-th category of the cropped image corresponding to the i-th sample of category c in the target image.

[0260] The present invention provides a model training system in which the feature determination module can be specifically used to: perform preliminary training on a target neural network model based on a target image and a label to obtain a preliminary trained target neural network model; input the target image into the preliminary trained target neural network model to obtain a class activation heatmap output by the preliminary trained target neural network model.

[0261] The model training system provided in this embodiment of the invention may further include:

[0262] The classification module is used to train the target neural network model for image classification based on the target image, target label, cropped image, and cropped label. After training the target neural network model, it obtains the image to be classified; transmits the image to be classified to the trained target neural network model; and obtains the classification result of the image to be classified output by the trained target neural network model.

[0263] This invention also provides an image cropping device and a computer-readable storage medium, both of which have the corresponding effects of the image cropping method provided in the embodiments of this invention. Please refer to... Figure 8 , Figure 8 This is a schematic diagram of the structure of an image cropping device provided in an embodiment of the present invention.

[0264] An image cropping device provided in this embodiment of the invention includes a memory 201 and a processor 202. The memory 201 stores a computer program, and the processor 202 executes the computer program to implement the steps of the image cropping method described in any of the above embodiments.

[0265] Please see Figure 9Another image cropping device provided in this embodiment of the invention may further include: an input port 203 connected to the processor 202 for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202 for displaying the processing results of the processor 202 to the outside; and a communication module 205 connected to the processor 202 for enabling communication between the image cropping device and the outside. The display unit 204 may be a display panel, a laser scanner, or the like; the communication method used by the communication module 205 includes, but is not limited to, Mobile High-Definition Link (MHL), Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), wireless connectivity: Wireless Fidelity (WiFi), Bluetooth communication technology, Bluetooth Low Energy communication technology, and communication technology based on IEEE 802.11s.

[0266] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image cropping method described in any of the above embodiments.

[0267] The computer-readable storage media involved in this invention include random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (compact disc read-only memory), or any other form of storage media known in the art.

[0268] For descriptions of relevant parts of the image acquisition system, device, and computer-readable storage medium provided in the embodiments of the present invention, please refer to the detailed description of the corresponding parts in the image acquisition method provided in the embodiments of the present invention, which will not be repeated here. Furthermore, parts of the technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0269] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0270] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image cropping method, characterized in that, include: Obtain the target image to be cropped and its corresponding target label; Determine the class activation features of the target image; Clustering is performed on the class activation features to obtain the clustering results; The target image is cropped to obtain a cropped image; Based on the clustering results, the semantic relevance between the cropped image and the target image is determined; Based on the semantic relevance and the target label, the cropping label of the cropped image is determined; The step of determining the semantic relevance between the cropped image and the target image based on the clustering results includes: Based on the clustering results, a first number of pixels belonging to the class-activated semantic region in the feature map is determined. The feature map includes a map obtained by feature extraction of the target image. Based on the clustering results, a second number of pixels belonging to the class-activated semantic region in the cropped image is determined; Using the bag-of-vocabularies model, a first semantic probability distribution of the feature map is determined based on the first quantity value; The second semantic probability distribution of the cropped image is determined based on the second quantity value using the bag-of-vocabulary model. The semantic relevance is determined based on the first quantity value, the second quantity value, the first semantic probability distribution, and the second semantic probability distribution.

2. The method according to claim 1, characterized in that, Determining the class activation features of the target image includes: Feature extraction is performed on the target image to obtain a feature map; The feature map is subjected to class activation to obtain a class activation heatmap; The activation heatmap is binarized to obtain a binary mask image; Perform an AND operation between the feature map and the binary mask image to obtain the AND result image; Based on the binary mask image and the AND operation result image, the class activation features are determined.

3. The method according to claim 2, characterized in that, The step of performing class activation on the feature map to obtain a class activation heatmap includes: The feature map is activated by class activation formula to obtain the class activation heatmap; The class activation formula includes: ； ； in, This represents the class activation heatmap; Indicates that the target image belongs to the category of The first sample feature extraction output The feature mapping map of each channel; Indicates the number of categories; Indicates the first The feature mapping map of each channel for each category The weights; Indicates category The output probability value for the first Pixels on the feature map of each channel gradient value, Indicates the first The total number of pixels on the feature map of each channel.

4. The method according to claim 2, characterized in that, The step of determining the class activation features based on the binary mask image and the AND operation result image includes: Determine the location of the pixel with the highest intensity value in the binary mask image; In the image resulting from the AND operation, the feature data corresponding to the pixel position is determined as the class activation feature.

5. The method according to claim 2, characterized in that, The process of clustering the activation features of the classes to obtain clustering results includes: The clustering results are obtained by clustering the class activation features based on the K-means clustering algorithm.

6. The method according to claim 2, characterized in that, The step of determining the first number of pixels belonging to the class-activated semantic region in the feature map based on the clustering result includes: For each first pixel in the feature map, in the clustering result, the first cluster center closest to the first pixel is determined, and a first distance value between the first pixel and the first cluster center is determined. If the first distance value is less than or equal to the cluster radius of the first cluster center, then the first pixel is marked as a pixel belonging to the class-activated semantic region in the feature map. The first count value of pixels belonging to the class-activated semantic region in the feature map is calculated.

7. The method according to claim 6, characterized in that, The step of determining the second number of pixels belonging to the class-activated semantic region in the cropped image based on the clustering result includes: For each second pixel in the cropped image, in the clustering result, the second cluster center closest to the second pixel is determined, and a second distance value between the second pixel and the second cluster center is determined. If the second distance value is less than or equal to the cluster radius of the second cluster center, then the second pixel is marked as a pixel belonging to the class-activated semantic region in the cropped image. The second count value of pixels belonging to the class-activated semantic region in the cropped image is calculated.

8. The method according to claim 7, characterized in that, The step of determining the first semantic probability distribution of the feature map based on the first quantity value using the bag-of-vocabulary model includes: For each cluster center in the clustering results, the third quantity value of the first pixel in the class-activated semantic region of the cluster center is calculated, and the ratio of the third quantity value to the first quantity value is used as the first semantic probability value corresponding to the cluster center. All the first semantic probability values ​​are combined to form a first probability distribution feature vector; The first probability distribution feature vector is used as the first semantic probability distribution; The step of determining the second semantic probability distribution of the cropped image based on the second quantity value using the bag-of-vocabulary model includes: For each cluster center in the clustering results, a fourth quantity value of the second pixel in the class-activated semantic region of the cluster center is calculated, and the ratio of the fourth quantity value to the second quantity value is used as the second semantic probability value corresponding to the cluster center. All the second semantic probability values ​​are combined to form a second probability distribution feature vector; The feature vector of the second probability distribution is used as the second semantic probability distribution.

9. The method according to claim 2, characterized in that, The step of determining the semantic relevance based on the first quantity value, the second quantity value, the first semantic probability distribution, and the second semantic probability distribution includes: The semantic region size similarity is determined based on the ratio of the second quantity value to the first quantity value; The semantic distribution similarity is determined based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution; The semantic region size similarity and the semantic distribution similarity are used as the semantic relevance.

10. The method according to claim 1, characterized in that, Determining the semantic region size similarity based on the ratio of the second quantity value to the first quantity value includes: The semantic region size similarity is determined based on the ratio of the second quantity value to the first quantity value using the semantic region size similarity calculation formula. The formula for calculating the similarity of semantic regions includes: ； in, Indicates that the target image belongs to the category of The The similarity in the size of the semantic regions of the samples; Indicates that the target image belongs to the category of The The second quantity value corresponding to each sample; Indicates that the target image belongs to the category of The The first quantity value corresponding to each sample; Indicates hyperparameters; Determining semantic distribution similarity based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution includes: The semantic distribution similarity is determined based on the cosine distance between the first semantic probability distribution and the second semantic probability distribution using the semantic distribution similarity calculation formula. The semantic distribution similarity calculation formula includes: ； in, Indicates that the target image belongs to the category of The The semantic distribution similarity corresponding to each sample; Indicates that the target image belongs to the category of The The first semantic probability distribution corresponding to each sample; Indicates that the target image belongs to the category of The The second semantic probability distribution corresponding to each sample; Indicates taking the absolute value; This represents hyperparameters.

11. The method according to claim 10, characterized in that, The target label includes the label of the target image itself and the predicted label of the target image by the target neural network model for image classification; Determining the cropping label of the cropped image based on the semantic relevance and the target label includes: The product of the semantic region size similarity and the tag is used as the pseudo-tag of the cropped image; The product of the semantic distribution similarity and the predicted label is used as the soft label of the cropped label; The pseudo-label and the soft label are used as the cropping label.

12. The method according to claim 1, characterized in that, Before determining the cropping label of the cropped image based on the semantic relevance and the target label, the method further includes: Determine whether the semantic relevance is greater than a preset threshold; If the semantic relevance is greater than the preset threshold, the cropped image is determined to be valid, and the step of determining the cropping label of the cropped image based on the semantic relevance and the target label is executed. If the semantic relevance is less than or equal to the preset threshold, the cropped image is determined to be invalid, and the process returns to cropping the target image to obtain the cropped image and subsequent steps.

13. The method according to claim 1, characterized in that, The process of obtaining the target image to be cropped includes: Obtain the original image to be cropped; The original image is preprocessed to obtain the target image; The preprocessing includes brightness normalization, noise filtering, scale normalization, and flipping.

14. A model training method, characterized in that, Includes the steps of the image cropping method according to any one of claims 1 to 13; Based on the target image, the target label, the cropped image, and the cropped label, a target neural network model for image classification is trained.

15. The method according to claim 14, characterized in that, The cropped labels include pseudo labels and soft labels; The training process of the target neural network model includes: The loss value is calculated based on the soft label and the pseudo label using the loss function calculation formula. The loss function calculation formula includes: ； ； ； in, This represents the calculated loss value; Indicates hyperparameters; Indicates the total number of categories; This indicates that the target neural network model is effective for the target image containing categories. The The cropped image corresponding to the sample is the first one. The predicted probability values ​​for each category; Indicates that the target image belongs to the category of The The cropped image corresponding to the sample is the first one. The values ​​of the pseudo-labels for each category; Indicates that the target image belongs to the category of The The cropped image corresponding to the sample of the nth sample The value of the soft label for each category.

16. The method according to claim 14, characterized in that, Feature extraction is performed on the target image to obtain a feature map. Class activation is then performed on the feature map to obtain a class activation heatmap, including: The target neural network model is initially trained based on the target image and the labeled data to obtain the initially trained target neural network model. The target image is input into the pre-trained target neural network model to obtain the class activation heatmap output by the pre-trained target neural network model.

17. The method according to claim 14, characterized in that, After training the target neural network model for image classification based on the target image, the target label, the cropped image, and the cropped label, the process further includes: Obtain the image to be classified; Transmit the image to be classified to the trained target neural network model; Obtain the classification result of the image to be classified from the output of the trained target neural network model.

18. An image cropping system, characterized in that, include: The image acquisition module is used to acquire the target image to be cropped and its corresponding target label; A feature determination module is used to determine the class activation features of the target image; The feature clustering module is used to cluster the class activation features to obtain clustering results; The image cropping module is used to crop the target image to obtain a cropped image; A semantic determination module is used to determine the semantic relevance between the cropped image and the target image based on the clustering results; A label determination module is used to determine the cropping label of the cropped image based on the semantic relevance and the target label; The semantic determination module is configured to: determine a first number of pixels belonging to a class-activated semantic region in a feature map based on the clustering results, wherein the feature map includes a map obtained by feature extraction of the target image; determine a second number of pixels belonging to a class-activated semantic region in the cropped image based on the clustering results; determine a first semantic probability distribution of the feature map based on the first number of pixels using a bag-of-vocabulary model; determine a second semantic probability distribution of the cropped image based on the second number of pixels using the bag-of-vocabulary model; and determine the semantic relevance based on the first number of pixels, the second number of pixels, the first semantic probability distribution, and the second semantic probability distribution.

19. An image cropping device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the image cropping method as described in any one of claims 1 to 13 when executing the computer program.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image cropping method as described in any one of claims 1 to 13.

Images