Classification and screening method and system based on semantic features

Abstract

The present application relates to the field of image classification, and in particular to a classification screening method and system based on semantic features, the method comprising: obtaining a trained classification network; inputting the image to be classified into the trained classification network to obtain a feature map output by each convolutional layer and a first classification result output by the classifier; obtaining the feature map and the classification contribution of each pixel in the image to be classified based on the first classification result; for any pixel in the image to be classified, selecting the target pixel corresponding to the maximum classification contribution as the classification significance point of the pixel; inputting the fused classification feature composed of the classification significance points of all pixels in the image to be classified into the classifier to obtain a second classification result. The present application improves the accuracy of the classification result by fusing the semantic feature with the largest classification contribution corresponding to each pixel in the image to be classified.

Description

Technical Field

[0001] The present application generally relates to the field of image classification, and more particularly to a classification and screening method and system based on semantic features. Background Art

[0002] With the rapid development of artificial intelligence technology, image classification is widely used in smart agriculture. In smart agriculture, classification tasks such as crop quality screening and crop maturity determination are involved. The accuracy of classification tasks directly affects crop quality and production efficiency.

[0003] At present, the patent document with publication number CN116385795A discloses an image classification method, which inputs the image to be classified into a pre-trained discriminative region learning dual-branch attention network for fine-grained image classification to obtain a classification result; wherein, in the framework of the dual-branch attention network, attention is erased on one of the branches to remove significant regional features, so as to better learn the semantic complementary information of the other branch; finally, through feature fusion, the multiple discriminative region features mined by the dual branches are fused to obtain the final image classification result of the image to be classified.

[0004] However, the above method learns the discriminative features that affect the classification results in the image to be classified by erasing. However, the semantic features of different scales of each pixel in the image to be classified have different significance for the classification results. Directly fusing the features of multiple discriminative regions will overwhelm the semantic features with higher significance for the classification results, resulting in inaccurate classification results. Summary of the Invention

[0005] In order to solve the above technical problems in the prior art, the present application provides a classification and screening method and system based on semantic features, thereby improving the accuracy of classification results.

[0006] The present invention provides a classification and screening method based on semantic features, comprising: obtaining a trained classification network, the classification network comprising a classifier and multiple convolutional layers; inputting an image to be classified into the trained classification network to obtain a feature map output by each convolutional layer and a first classification result output by the classifier, wherein a pixel point in the image to be classified corresponds to multiple target pixels in the feature map, and the pixel values of different target pixels are used to reflect the semantic features of the pixel point at different scales in the image to be classified; obtaining the feature map and the classification contribution of each pixel point in the image to be classified based on the first classification result; for any pixel point in the image to be classified, selecting the target pixel point corresponding to the maximum classification contribution as the classification significance point of the pixel point; and inputting a fused classification feature composed of the classification significance points of all pixels in the image to be classified into the classifier to obtain a second classification result.

[0007] In some embodiments, the classifier includes a 1×1×N convolutional layer, a global pooling layer and a Softmax layer; the 1×1×N convolutional layer is used to convert the number of channels of a feature map of any size into N, where N is the number of categories; the global pooling layer is used to perform a global average pooling operation on the output result of the 1×1×N convolutional layer, and output a feature vector of 1 row and N columns; the Softmax layer normalizes the feature vector to output a category vector, which includes the probability that the feature map belongs to each category.

[0008] In some embodiments, the first classification result includes a first probability that the image to be classified belongs to each category. Before obtaining the classification contribution of the feature map and each pixel in the image to be classified based on the first classification result, the method further includes: calculating the information entropy of the first classification result; in response to the information entropy being less than a set entropy value, using the first classification result as the second classification result; in response to the information entropy being not less than the set entropy value, screening at least two candidate categories based on the first classification result.

[0009] In some embodiments, screening at least two candidate categories based on the first classification result includes: arranging all first probabilities in the first classification result in descending order, and then determining the number of candidate categories based on a set judgment condition, wherein the set judgment condition satisfies the relationship:

[0010]

[0011] Among them, p1(1) and p1(m) are the first probabilities ranked in the first and mth positions respectively, N is the number of all first probabilities, φ is the set threshold, and m that meet the set judgment conditions is the number of candidate categories; the category types corresponding to the first m first probabilities are taken as candidate categories.

[0012] In some embodiments, obtaining the classification contribution of each pixel in the feature map and the image to be classified based on the first classification result includes: taking the image to be classified or a feature map as a target image, the target image including at least one image channel; calculating a gradient image of a first probability of a candidate category to the image channel, the gradient image including the gradient value of each pixel in the image channel; taking the mean of the gradient values in the gradient image as the weight coefficient of the corresponding image channel; performing weighted summation on the gradient images of all image channels according to the weight coefficient to obtain a contribution subgraph of the candidate category to the target image, the contribution subgraph satisfying the relationship:

[0013]

[0014] Wherein, j represents the target image, Num(j) is the number of image channels in the target image j, is the weight coefficient of image channel k in the target image j, is the gradient image of image channel k in the target image j, is the contribution subgraph of candidate category a to the target image j; based on the contribution subgraphs of all candidate categories to the target image, a classification contribution graph of the target image is determined, wherein the classification contribution graph includes the classification contribution of each pixel in the target image, and the classification contribution graph satisfies the relationship:

[0015]

[0016] Where m is the number of all candidate categories, p1(a) is the first probability corresponding to candidate category a, and Sum(p1) is the sum of the first probabilities corresponding to all candidate categories. is the contribution subgraph of candidate category a to the target image j, L j is the classification contribution map of the target image j.

[0017] In some embodiments, the second classification result includes a second probability that the image to be classified belongs to each category. After obtaining the second classification result, the method further includes: among all candidate categories, taking the candidate category corresponding to the maximum second probability as the category to which the image to be classified belongs.

[0018] In some embodiments, the training method of the classification network includes: collecting image information with category labels, where the category labels are the category types to which the corresponding image information belongs; building an initial classification network, where the initial classification network includes an initial classifier and an initial encoder, and the initial encoder includes multiple initial convolutional layers; inputting an image information into the initial classification network to obtain multiple feature maps and a first classification result; obtaining the classification significance points of each pixel point in the image information, and inputting the fused classification features composed of all classification significance points into the initial classifier to obtain a second classification result; calculating the loss function value based on the second classification result and the category label, and using the gradient descent method to update the network parameters of the initial classifier and multiple initial convolutional layers to reduce the loss function value; iteratively updating the network parameters of the initial classifier and multiple initial convolutional layers until the loss function value is less than the set loss, or the number of iterations is greater than the set number, to obtain a trained classification network.

[0019] In some embodiments, the method for obtaining the fused classification feature includes: replacing each pixel point in the image to be classified with the pixel value of the corresponding classification salient point to obtain the fused classification feature.

[0020] In some embodiments, the method for obtaining the fused classification features includes: for the image to be classified or a feature map, retaining the pixel values at the classification significance points, setting the pixel values of the pixels other than the classification significance points to 0, and obtaining the classification significance map corresponding to the image to be classified or the feature map; after adjusting all the classification significance maps to a preset size, adding the pixel values at the corresponding positions to obtain the fused classification features.

[0021] The present application also provides a classification and screening system based on semantic features, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the classification and screening method based on semantic features described in the first aspect of the present application is implemented.

[0022] The above-mentioned classification and screening method based on semantic features provided in the embodiment of the present application first inputs the image to be classified into a trained classification network to obtain feature maps of different sizes and a first classification result, and further obtains the classification contribution of the image to be classified and the pixel points in each feature map based on the gradient information of the first classification result; for any pixel point in the image to be classified, the most effective semantic feature of the pixel point for classification is screened from the image to be classified and the feature map based on the classification contribution, and is used as the classification significance point of the pixel point, and the second classification result is obtained based on the features of all classification significance points. By fusing the semantic feature with the greatest classification contribution of each pixel point in the image to be classified, the accuracy of the classification result is improved.

[0023] Furthermore, each pixel in the image to be classified corresponds to multiple target pixels, and different target pixels are located on the feature maps output by different convolutional layers, and the pixel values of different target pixels can be used to represent the semantic features of the pixels at different scales in the image to be classified; the classification significance point of each pixel in the image to be classified is determined based on the classification contribution of the target pixel, and the pixel value of the classification significance point can be used to represent the semantic feature of the corresponding pixel that is most effective for the classification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the exemplary embodiments of the present application will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0025] Figure 1 is a flow chart of a classification and screening method based on semantic features according to an embodiment of the present application;

[0026] Figure 2 2 is a schematic diagram of a convolution operation performed by the first convolution layer according to an embodiment of the present application;

[0027] Figure 3 2 is a schematic diagram of a convolution operation performed by the second convolution layer according to an embodiment of the present application;

[0028] Figure 4 is a flowchart of a training method for a classification network according to an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of training an initial classification network according to an embodiment of the present application;

[0030] Figure 6 4 is a block diagram of a classification and screening system based on semantic features according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0032] It should be understood that when the terms "first," "second," etc. are used in the claims, specification, and drawings of this application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "comprise" and "comprising" used in the specification and claims of this application indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0033] According to the first aspect of the present application, the present application provides a classification and screening method based on semantic features. Figure 1 FIG. 1 is a flowchart of a classification and screening method based on semantic features according to an embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0034] S11, obtaining a trained classification network, where the classification network includes a classifier and multiple convolutional layers.

[0035] In one embodiment, the classification network includes an encoder and a classifier. The encoder includes multiple convolutional layers, each of which is used to perform a convolution operation on the feature map output by the previous convolutional layer to obtain the feature map output by the current convolutional layer, wherein the size and step size of the convolution kernel in the convolution operation are preset. Specifically, the encoder can adopt the encoding structure of existing convolutional neural networks such as VGG, ResNet, or MobileNet.

[0036] To enable the classifier to process feature maps of any size, it includes a 1×1×N convolutional layer, a global pooling layer, and a Softmax layer. The 1×1×N convolutional layer converts the number of channels of a feature map of any size to N, where N is the number of categories. The global pooling layer performs a global average pooling operation on the output of the 1×1×N convolutional layer, outputting a 1-row, N-column feature vector. The Softmax layer normalizes the feature vector to output a category vector containing the probability that the feature map belongs to each category. The sum of all values in the category vector is 1.

[0037] Exemplarily, the encoder uses the VGG16 encoding structure and sets the number of categories to 10. Specifically, the encoder includes 13 convolutional layers, and the classifier includes a 1×1×10 convolutional layer, a global pooling layer, and a Softmax layer. When the image to be classified is an RGB image of size 224×224×3, the feature map output by the last convolutional layer in the encoder has a size of 24×24×512, where 3 and 512 represent the number of image channels. After processing by the 1×1×10 convolutional layer in the classifier, the feature map size becomes 24×24×10. This 24×24×10 feature map is then fed into the global pooling layer to produce a feature vector with one row and ten columns. The Softmax layer normalizes this feature vector to produce a category vector.

[0038] S12, inputting the image to be classified into the trained classification network to obtain the feature map output by each convolution layer and the first classification result output by the classifier, wherein one pixel point in the image to be classified corresponds to multiple target pixels in the feature map, and the pixel values of different target pixels are used to reflect the semantic features of the pixel points at different scales in the image to be classified.

[0039] In one embodiment, for a trained classification network, after inputting an image to be classified, each convolutional layer performs a convolution operation on the feature map output by the previous convolutional layer to obtain a feature map output by each convolutional layer; the feature map output by the last convolutional layer is input into the classifier to obtain a first classification result, wherein the first classification result includes a first probability that the image to be classified belongs to each category.

[0040] When multiple convolutional layers perform convolution operations on the image to be classified, the receptive field of the feature map corresponding to each convolutional layer gradually increases as the number of convolutional layers increases. In other words, the larger the number of convolutional layers, the larger the rectangular area in the image to be classified that a pixel in the feature map is mapped to, and the larger the scale of the semantic features that a pixel in the feature map can reflect. Semantic features are comprehensive features within the rectangular area of the image to be classified. For example, for a pixel (x, y), the comprehensive features within a 3×3 area centered at (x, y) are the 3×3 semantic features of the pixel (x, y); the comprehensive features within a 7×7 area centered at (x, y) are the 7×7 semantic features of the pixel (x, y).

[0041] Specifically, the encoder uses the VGG16 encoding structure; first, the periphery of the image to be classified is filled. The specific filling method can be zero filling or directly copying the edge pixels of the image to be classified. This application does not impose any restrictions. Figure 2 , is a schematic diagram of the convolution operation performed by the first convolution layer according to an embodiment of the present application. The first convolution layer slides on the image to be classified using a 3×3 convolution kernel with a step size of 1; for each pixel in the image to be classified, its corresponding semantic feature within the 3×3 scale range can be found in the output feature map of the first convolution layer. In other words, a pixel in the image to be classified corresponds to a target pixel in the output feature map of the first convolution layer, and the pixel value of the target pixel can be used to reflect the semantic feature of the pixel within the 3×3 scale range in the image to be classified.

[0042] For further information, see Figure 3 , is a schematic diagram of the convolution operation performed by the second convolution layer according to an embodiment of the present application. The second convolution layer uses a 3×3 convolution kernel with a step size of 2 to slide on the output feature map of the first convolution layer; for each pixel in the output feature map of the first convolution layer, its corresponding semantic feature within the 3×3 scale range can be found in the output feature map of the second convolution layer; but since the step size of the convolution kernel in the second convolution layer is 2, a pixel in the output feature map of the second convolution layer corresponds to a 7×7 area in the image to be classified. In other words, a pixel in the image to be classified corresponds to a target pixel in the output feature map of the second convolution layer, and the pixel value of the target pixel can be used to reflect the semantic feature of the pixel within the 7×7 scale range in the image to be classified.

[0043] According to the same method, all target pixels corresponding to a pixel in the image to be classified in the feature map can be determined, and all target pixels are located on the feature maps output by different convolutional layers. All target pixels can be used to represent the semantic features of the corresponding pixels at different scales. As the number of convolutional layers increases, the scale of the semantic features represented by the pixel value of the target pixel becomes larger.

[0044] In this way, after the image to be classified is input into the trained classification network, multiple target pixels corresponding to each pixel in the image to be classified can be obtained. Different target pixels are located on the feature maps output by different convolutional layers, and the pixel values of different target pixels can be used to represent the semantic features of pixels at different scales in the image to be classified.

[0045] S13: Obtain the classification contribution of each pixel in the feature map and the image to be classified based on the first classification result.

[0046] In one embodiment, the first classification result includes a first probability that the image to be classified belongs to each category. Before obtaining the classification contribution of the feature map and each pixel in the image to be classified based on the first classification result, the method further includes: calculating the information entropy of the first classification result; in response to the information entropy being less than a set entropy value, using the first classification result as the second classification result; and in response to the information entropy being not less than the set entropy value, screening at least two candidate categories based on the first classification result.

[0047] The calculation formula of the information entropy is:

[0048]

[0049] in, is the first probability of category i in the first classification result, N is the number of categories, and ρ is the information entropy of the first classification result. A larger information entropy indicates greater information confusion in the first classification result, making it less likely to obtain an accurate classification result. The entropy value is set to 0.2. When the information entropy is less than the set entropy value, the first classification result is accurate and is used as the second classification result.

[0050] In one embodiment, screening at least two candidate categories based on the first classification result includes: arranging all first probabilities in the first classification result in descending order, and then determining the number of candidate categories based on a set judgment condition, wherein the set judgment condition satisfies the relationship:

[0051]

[0052] Where p1(1) and p1(m) are the first probabilities ranked at the first and mth positions, respectively; N is the number of all first probabilities; φ is the set threshold; the number of m candidates that meet the set judgment condition is the number of candidate categories; the categories corresponding to the first m first probabilities are selected as candidate categories. The value of φ is 0.7.

[0053] For example, assume that the first classification result includes a total of 5 categories, and the first probability of each category is arranged in order from large to small, namely 0.4, 0.3, 0.2, 0.1, and 0; its information entropy is 0.556, which is greater than the set entropy value of 0.2, and 0.4+0.3+0.2>0.7; therefore, the categories corresponding to the first probabilities of 0.4, 0.3, and 0.2 are taken as candidate categories.

[0054] It can be understood that the candidate category is the category to which the image to be classified may belong, which is screened according to the first classification result, and the category to which the image to be classified belongs is one of the multiple candidate categories.

[0055] In one embodiment, after obtaining the candidate categories, the classification contribution of each pixel in the feature map and the image to be classified can be determined based on the gradient information of the first probability corresponding to each candidate category. The classification contribution is used to characterize the degree of influence of the characteristics of the pixel on the classification result, which is specifically described as follows. The method of obtaining the classification contribution of each pixel in the feature map and the image to be classified based on the first classification result includes: taking the image to be classified or a feature map as the target image, the target image includes at least one image channel; calculating the gradient image of the first probability of a candidate category to the image channel, the gradient image includes the gradient value of each pixel in the image channel; taking the mean of the gradient values in the gradient image as the weight coefficient of the corresponding image channel; performing weighted summation on the gradient images of all image channels according to the weight coefficient to obtain a contribution subgraph of the candidate category to the target image, and the contribution subgraph satisfies the relationship:

[0056]

[0057] Wherein, j represents the target image, Num(j) is the number of image channels in the target image j, is the weight coefficient of image channel k in the target image j, is the gradient image of image channel k in the target image j, is the contribution subgraph of candidate category a to the target image j; based on the contribution subgraphs of all candidate categories to the target image, a classification contribution graph of the target image is determined, wherein the classification contribution graph includes the classification contribution of each pixel in the target image, and the classification contribution graph satisfies the relationship:

[0058]

[0059] Where m is the number of all candidate categories, p1(a) is the first probability corresponding to candidate category a, and Sum(p1) is the sum of the first probabilities corresponding to all candidate categories. is the contribution subgraph of candidate category a to the target image j, L j is the classification contribution map of the target image j.

[0060] It can be understood that the contribution subgraph of candidate category a to the target image In the graph, the pixel value of each pixel is used to reflect the classification contribution of its own features to the candidate category a. In other words, the contribution subgraph It can reflect which pixel features in the target image the classification network uses to determine the candidate class a. Similarly, the classification contribution graph can reflect which pixel features in the target image the classification network uses to determine all candidate classes.

[0061] In this way, multiple candidate categories corresponding to the image to be classified are determined based on the first classification result, and the classification contribution of each feature map and each pixel point in the image to be classified is obtained based on the gradient information of each candidate category. The classification contribution is used to reflect the degree of influence of the characteristics of the corresponding pixel point on the classification.

[0062] S14: For any pixel point in the image to be classified, select a target pixel point corresponding to the maximum classification contribution as a classification salient point of the pixel point.

[0063] In one embodiment, each pixel in the image to be classified corresponds to a plurality of target pixels, and the pixel values of the plurality of target pixels are used to reflect the semantic features of the corresponding pixel at different scales in the image to be classified. For the pixel (x, y) in the image to be classified, the classification contribution of the target pixel is obtained, and the target pixel corresponding to the maximum classification contribution is used as the classification significance point of the pixel (x, y). The pixel value of the classification significance point is used to characterize the most effective semantic feature of the pixel (x, y) for the classification result. The pixel value of the classification significance point is a vector of 1 row and D columns, and D is the number of image channels of the feature map where the classification significance point is located. For example, the image to be classified is an image including three image channels of RGB. When the classification significance point is in the image to be classified, the pixel value of the classification significance point is a vector of 1 row and 3 columns.

[0064] For example, the classification saliency point of the pixel point (x, y) corresponds to the semantic feature within a 7×7 area in the image to be classified, and the semantic feature representing the pixel point (x, y) within the 7×7 area is most effective for the classification result.

[0065] In this way, the classification salient point of each pixel in the image to be classified is obtained, and the pixel value of the classification salient point can be used to represent the most effective semantic feature of the corresponding pixel for the classification result. The scales of the semantic features corresponding to different pixels are the same or different.

[0066] S15, inputting the fused classification features composed of the classification saliency points of all pixels in the image to be classified into the classifier to obtain a second classification result.

[0067] In an optional embodiment, since the classification salient points corresponding to different pixels are selected from the feature maps output by the same or different convolutional layers, the number of columns of pixel values of the classification salient points is different. After obtaining the classification salient points for all pixels in the image to be classified, the pixel values of all classification salient points are fused to obtain a fused classification feature, as described in detail below. The method for obtaining the fused classification feature includes replacing each pixel in the image to be classified with the pixel value of the corresponding classification salient point to obtain the fused classification feature.

[0068] The size of the fused classification feature is W×H×max(D), where W×H is the size of the image to be classified, and max(D) is the maximum number of columns in the pixel values corresponding to all classification salient points. It can be understood that if the number of columns of pixel values corresponding to a classification salient point is less than max(D), zeros are padded until the number of columns equals max(D).

[0069] In another optional embodiment, the classification salient points are pixels in the image to be classified or any feature map. The method for obtaining the fused classification feature includes: for the image to be classified or a feature map, retaining the pixel values at the classification salient points and setting the pixel values of pixels other than the classification salient points to 0, thereby obtaining a classification salient map corresponding to the image to be classified or the feature map; and after adjusting all classification salient maps to a preset size, adding the pixel values at corresponding positions to obtain the fused classification feature.

[0070] The preset size is max(w)×max(h)×max(c), where max(w) is the maximum width of all images containing categorical salient points, max(h) is the maximum height of all images containing categorical salient points, and max(c) is the maximum number of image channels of all images containing categorical salient points. Resizing all categorical saliency maps to the preset size can be implemented using the resize function in OpenCV.

[0071] In one embodiment, the fused classification features are input into a classifier to obtain a second classification result, which is a classification result obtained based on the features of the classification salient points corresponding to all pixels in the image to be classified; the second classification result includes a second probability that the image to be classified belongs to each category. After obtaining the second classification result, the method further includes, among all candidate categories, selecting the candidate category corresponding to the maximum second probability as the category to which the image to be classified belongs.

[0072] In other embodiments, among all categories, the category corresponding to the second maximum probability value is used as the category to which the image to be classified belongs.

[0073] In this way, the second classification result is determined based on the most effective semantic feature of each pixel in the image to be classified for the classification result, the semantic features of each pixel at different scales are fully integrated, the features that are instructive for classification are extracted, and the accuracy of the classification result is improved.

[0074] In one embodiment, see Figure 4 , is a flowchart of a training method for a classification network according to an embodiment of the present application.

[0075] Please parameter Figure 5 , is a training diagram of the initial classification network according to an embodiment of the present application. The training method of the classification network includes: S21, collecting image information with category labels, the category labels are the category types to which the corresponding image information belongs; S22, building an initial classification network, the initial classification network includes an initial classifier and an initial encoder, and the initial encoder includes multiple initial convolution layers; S23, inputting an image information into the initial classification network to obtain multiple feature maps and a first classification result; S24, obtaining the classification significance points of each pixel in the image information, and inputting the fusion classification features composed of all classification significance points into the initial classifier to obtain a second classification result; S25, calculating the loss function value based on the second classification result and the category label, and using the gradient descent method to update the network parameters of the initial classifier and multiple initial convolution layers to reduce the loss function value; S26, iteratively updating the network parameters of the initial classifier and multiple initial convolution layers until the loss function value is less than the set loss, or the number of iterations is greater than the set number, to obtain a trained classification network. In one example, the set loss is 0.001 and the set number is 5000. Understandably, Figure 5 The initial classifiers whose parameters are shared by the two are the same initial classifier.

[0076] The initial classification network has the same network structure as the classification network, that is, the initial classifier has the same structure as the classifier, and the multiple initial convolutional layers have the same structure as the multiple convolutional layers. The network parameters in the initial classifier are the parameters of the convolution kernel in the 1×1×N convolutional layer, and the network parameters of the multiple initial convolutional layers are the parameters of the convolution kernel in the convolutional layer.

[0077] The process of obtaining the second classification result in the initial classification network is the same as the process of obtaining the second classification result in the trained classification network, and will not be described in detail here. The loss function value is a cross entropy loss function.

[0078] In this way, a trained classification network is obtained, and a second classification result of the image to be classified can be obtained based on the trained classification network.

[0079] In one application scenario, potatoes are graded and screened for quality. The image to be classified is any potato image. The output second classification result includes a second probability that the potato image belongs to each quality grade. The quality grade corresponding to the maximum second probability is used as the quality grading result of the potato.

[0080] The above-mentioned classification and screening method based on semantic features provided in the embodiment of the present application first inputs the image to be classified into a trained classification network to obtain feature maps of different sizes and a first classification result, and further obtains the classification contribution of the image to be classified and the pixel points in each feature map based on the gradient information of the first classification result; for any pixel point in the image to be classified, the most effective semantic feature of the pixel point for classification is screened from the image to be classified and the feature map based on the classification contribution, and is used as the classification significance point of the pixel point, and the second classification result is obtained based on the features of all classification significance points. By fusing the semantic feature with the greatest classification contribution of each pixel point in the image to be classified, the accuracy of the classification result is improved.

[0081] According to the second aspect of the present application, the present application also provides a classification and screening system based on semantic features. Figure 6 is a block diagram of a classification and screening system based on semantic features according to an embodiment of the present application. Figure 6 As shown, the apparatus 50 includes a processor and a memory. The memory stores computer program instructions. When the computer program instructions are executed by the processor, the semantic feature-based classification and screening method according to the first aspect of the present application is implemented. The apparatus also includes other components familiar to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are well known in the art and are therefore not described in detail here.

[0082] In the present application, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc., or any other medium that can be used to store the required information and can be accessed by an application, module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in this application may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0083] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A classification and screening method based on semantic features, characterized by: Obtain the trained classification network, which includes a classifier and multiple convolutional layers; Inputting the image to be classified into the trained classification network to obtain a feature map output by each convolutional layer and a first classification result output by the classifier, wherein one pixel in the image to be classified corresponds to multiple target pixels in the feature map, and the pixel values of different target pixels are used to reflect semantic features of the pixel at different scales in the image to be classified. The first classification result includes a first probability that the image to be classified belongs to each category. Calculate the information entropy of the first classification result; In response to the information entropy being less than the set entropy value, taking the first classification result as the second classification result; In response to the information entropy being not less than a set entropy value, screening at least two candidate categories based on the first classification result; Obtaining the classification contribution of each pixel in the feature map and the image to be classified based on the first classification result, including: The image to be classified or a feature map is used as the target image. The target image includes at least one image channel. A pixel in the image to be classified corresponds to different target pixels on the feature maps output by different convolutional layers. Different target pixels are used to represent semantic features of different scales reflected by the pixel in the image to be classified. Calculate a gradient image of a first probability of a candidate category with respect to an image channel, where the gradient image includes a gradient value of each pixel in the image channel; The mean of the gradient values in the gradient image is used as the weight coefficient of the corresponding image channel; The gradient images of all image channels are weighted and summed according to the weight coefficient to obtain the contribution subgraph of the candidate category to the target image. The contribution subgraph satisfies the relationship: Among them, j represents the target image, For the target image The number of image channels in , For the target image Middle image channel The weight coefficient of For the target image Middle image channel The gradient image, Candidate category For the target image Contribution subgraph of ; The classification contribution graph of the target image is determined based on the contribution subgraphs of all candidate categories to the target image. The classification contribution graph includes the classification contribution of each pixel in the target image. The classification contribution graph satisfies the relationship: in, is the number of all candidate categories, Candidate category The corresponding first probability is, is the sum of the first probabilities corresponding to all candidate categories, Candidate category For the target image The contribution subgraph of For the target image Classification contribution diagram of ; For any pixel in the image to be classified, the target pixel corresponding to the maximum classification contribution is selected as the classification salient point of the pixel. The classification salient points corresponding to different pixels are selected from the feature maps output by the same or different convolutional layers. The pixel value of the classification salient point is used to represent the semantic feature that is most effective for the classification result of the corresponding pixel. In the image to be classified, each pixel is replaced by the pixel value of the corresponding classification salient point to obtain a fused classification feature; the fused classification feature composed of the classification salient points of all pixels in the image to be classified is input into the classifier to obtain the second classification result.

2. A classification and screening method based on semantic features as claimed in claim 1, characterized in that: The classifier includes a 1×1×N convolutional layer, a global pooling layer and a Softmax layer; The 1×1×N convolutional layer is used to convert the number of channels of feature maps of any size into N, where N is the number of categories; The global pooling layer is used to perform a global average pooling operation on the output result of the 1×1×N convolutional layer, and output a feature vector of 1 row and N columns; The Softmax layer normalizes the feature vector to output a category vector, which includes the probability that the feature map belongs to each category.

3. A classification and screening method based on semantic features as claimed in claim 2, characterized in that: The screening of at least two candidate categories based on the first classification result includes: After arranging all the first probabilities in the first classification results in descending order, the number of candidate categories is determined according to a set judgment condition, and the set judgment condition satisfies the relationship: in, and are the first probabilities of being ranked in the first and mth positions respectively, is the number of all first probabilities, is to set a threshold, and m that meets the set judgment condition is the number of candidate categories; The categories corresponding to the first m probabilities are taken as candidate categories.

4. A classification and screening method based on semantic features as claimed in claim 3, characterized in that: The second classification result includes a second probability that the image to be classified belongs to each category. After obtaining the second classification result, the method further includes: Among all candidate categories, the candidate category corresponding to the second maximum probability value is used as the category to which the image to be classified belongs.

5. The classification and screening method based on semantic features according to claim 2, wherein: The training method of the classification network includes: Collect image information with category labels, where the category labels are the categories to which the corresponding image information belongs; Building an initial classification network, the initial classification network includes an initial classifier and an initial encoder, and the initial encoder includes multiple initial convolutional layers; Inputting image information into the initial classification network to obtain multiple feature maps and a first classification result; Obtaining a classification salient point for each pixel in the image information, and inputting a fusion classification feature composed of all the classification salient points into the initial classifier to obtain a second classification result; Calculating a loss function value based on the second classification result and the category label, and updating network parameters of the initial classifier and multiple initial convolutional layers using a gradient descent method to reduce the loss function value; Iteratively update the network parameters of the initial classifier and multiple initial convolutional layers until the loss function value is less than the set loss, or the number of iterations is greater than the set number, to obtain a trained classification network.

6. The classification and screening method based on semantic features according to claim 1, wherein: The method for obtaining the fused classification features includes: For the image to be classified or a feature map, retain the pixel values at the classification significance points, set the pixel values of the pixels other than the classification significance points to 0, and obtain a classification significance map corresponding to the image to be classified or the feature map; After adjusting all classification saliency maps to the preset size, the pixel values at the corresponding positions are added to obtain the fused classification features.

7. A classification and screening system based on semantic features, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the classification and screening method based on semantic features according to any one of claims 1 to 6 is implemented.

Images