Image classification method, device, terminal device and storage medium

Through the two-stage image classification method, the capsule endoscopic image is deformed using significant images, and the lesion area characteristics are amplified, which solves the problem of low accuracy in the classification of capsule endoscopic image in the prior art, which significantly improves the recognition of lesion images accuracy.

CN112634231BActive Publication Date: 2024-11-19SHENZHEN RES INST THE CHINESE UNIV OF HONG KONG
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202011539202.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-23
Publication Date
2024-11-19
Estimated Expiration
2040-12-23

AI Technical Summary

Technical Problem

The existing capsule endoscopic image classification accuracy is low, mainly due to the limited number of images in the data set and the indistinguishability of the lesion region and the normal region characteristics.

Method used

Using a two-stage image classification method, first obtain significant images through the first classification network and deform the image, enlarge the target area with a larger significant value, and then input the deformed image into the second classification network for identification, and finally combine the two networks. The prediction results determine the image classification results.

Benefits of technology

By amplifying the characteristics of the lesion area, the accuracy of recognition of the lesion image is improved and the accuracy of capsule endoscopic image classification is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112634231B_ABST
    Figure CN112634231B_ABST
Patent Text Reader

Abstract

The present application relates to the field of image processing technology, and proposes an image classification method, apparatus, terminal device and computer storage medium. After acquiring an image to be classified, the present application inputs the image to be classified into a classification network to obtain a corresponding saliency image and a category prediction result, and the saliency image marks the saliency value of each position in the image to be classified. Then, the image to be classified is deformed according to the saliency map, so as to obtain a deformed image that enlarges the target position in the image. Next, another classification network is used to identify the deformed image to obtain another category prediction result, and finally the two category prediction results are combined to obtain the final image classification result. By setting it in this way, the accuracy of image classification can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image classification method, apparatus, terminal device and storage medium. Background Art

[0002] Capsule endoscope is an endoscope in the shape of a capsule, which is used to observe the health status of the gastrointestinal tract, esophagus and other parts of the human body, helping doctors to make accurate diagnosis of patients.

[0003] At present, for the processing of capsule endoscopy images, an image classification model with a neural network structure is usually used to classify the images to distinguish normal images from pathological images.

[0004] However, due to the limited number of images in the existing capsule endoscopy dataset and the difficulty in distinguishing between the features of the lesion area and the normal area, the accuracy of capsule endoscopy image classification is low. Summary of the invention

[0005] In view of this, embodiments of the present application provide an image classification method, apparatus, terminal device and storage medium, which can improve the accuracy of image classification.

[0006] A first aspect of an embodiment of the present application provides an image classification method, comprising:

[0007] Get the image to be classified;

[0008] Inputting the image to be classified into a first classification network to obtain a first category prediction result of the image to be classified and a saliency map of the image to be classified, wherein the saliency map includes saliency values ​​of various positions in the image to be classified;

[0009] Performing image deformation processing on the image to be classified according to the saliency map to obtain a deformed image that enlarges a target position in the image to be classified, wherein the target position is a position in the image to be classified where the saliency value is greater than a specified value;

[0010] Inputting the deformed image into a second classification network to obtain a second category prediction result of the image to be classified;

[0011] The classification result of the image to be classified is determined according to the first category prediction result and the second category prediction result.

[0012] After acquiring the image to be classified, the embodiment of the present application inputs the image to be classified into a classification network to obtain a corresponding saliency image and a category prediction result. The saliency image marks the saliency value of each position in the image to be classified. Then, the image to be classified is deformed according to the saliency map to obtain a deformed image that enlarges the target position in the image. Next, another classification network is used to identify the deformed image to obtain another category prediction result, and finally the two category prediction results are combined to obtain the final image classification result. Taking the recognition of lesion images as an example, since the saliency value of the lesion area in the lesion image is large, a deformed image that enlarges the lesion area in the image will be obtained, and by enlarging the lesion area, the lesion features in the image will also correspondingly make a greater contribution to the features of the entire image, so the accuracy of lesion image recognition can be improved.

[0013] In one embodiment of the present application, the first classification network has a plurality of convolution modules, and inputting the image to be classified into the first classification network to obtain a saliency map of the image to be classified may include:

[0014] After the image to be classified is input into the first classification network, image features of the image to be classified are extracted by the multiple convolution modules;

[0015] A saliency map of the image to be classified is constructed according to feature values ​​corresponding to the image features at various positions of the image to be classified.

[0016] The convolution module can be used to extract image features, and according to the feature values ​​of the image features at various positions in the image, the saliency value of each position in the image can be determined, thereby obtaining a saliency map of the image.

[0017] In one embodiment of the present application, performing image deformation processing on the image to be classified according to the saliency map to obtain a deformed image in which a target position in the image to be classified is enlarged may include:

[0018] Constructing a sampling point distribution image of the image to be classified according to the saliency map, wherein the sampling point distribution image records each sampling point distributed in the image to be classified, and the sampling point distribution density at each position in the image to be classified is proportional to the saliency value of the position;

[0019] The image to be classified is sampled according to the sampling point distribution image to obtain the deformed image.

[0020] When deforming the image to be classified, the sampling density of the position is determined according to the saliency value of the different positions in the image. The target area with a larger saliency value has a denser sampling; on the contrary, the sampling will be very sparse in the general area with a smaller saliency value. The deformed image will enlarge the target area and compress the general area.

[0021] Furthermore, constructing a sampling point distribution image of the image to be classified according to the saliency map may include:

[0022] Generate an initial image with uniformly distributed sampling points according to the size of the image to be classified;

[0023] Based on the saliency values ​​of each position in the image to be classified contained in the saliency map, a convolution kernel of Gaussian distribution is used to perform convolution processing on the initial image to obtain the sampling point distribution image.

[0024] When constructing the sampling point distribution image based on the saliency map, first, an initial image with uniformly distributed sampling points can be generated according to the size of the image to be classified, that is, the initial sampling field; then, the convolution kernel of the Gaussian distribution is used to convolve the initial sampling field with uniform distribution, in which each sampling point will pull the surrounding sampling points with a force proportional to its saliency value, so the target area with high saliency value will attract the surrounding sampling points with a greater force, thereby generating densely distributed sampling points. Conversely, the area with low saliency value has a relatively small pulling force on the surrounding sampling points, so the generated sampling points are sparsely distributed.

[0025] In one embodiment of the present application, the image classification method may further include:

[0026] Calculating the loss function of the second classification network according to the second category prediction result;

[0027] Optimizing the parameters of the first classification network according to the loss function of the second classification network.

[0028] The parameters of the two classification networks can be trained and optimized, and can be trained in an end-to-end manner. If the saliency map generated by the first classification network is inaccurate, the image input to the second classification network will not achieve the desired deformation, resulting in low recognition accuracy of the second classification network and a large loss function. The loss function will generate a gradient back-propagation to the first classification network to correct and optimize the parameters of the first classification network.

[0029] In one embodiment of the present application, the image classification method may further include:

[0030] Calculating a loss function of the first classification network according to the first category prediction result;

[0031] Optimizing parameters of the second classification network according to the loss function of the first classification network.

[0032] After obtaining the prediction result of the first category, the loss function of the first classification network can be calculated according to the prediction result of the first category, and then the parameters of the second classification network can be corrected and optimized by using the gradient back propagation method in combination with the loss function of the first classification network, thereby improving the performance of the second classification network.

[0033] In one embodiment of the present application, the first category prediction result includes a probability prediction value of each image category in a plurality of preset image categories, and the second category prediction result also includes a probability prediction value of each image category in the plurality of image categories. Determining the classification result of the image to be classified according to the first category prediction result and the second category prediction result may include:

[0034] For each image category in the multiple image categories, respectively calculate an average of a probability prediction value of the image category in the first category prediction result and a probability prediction value of the image category in the second category prediction result to obtain a target probability prediction value of each image category in the multiple image categories;

[0035] The image category with the largest target probability prediction value among the multiple image categories is determined as the classification result of the image to be classified.

[0036] For example, if there are K image categories in total, a K-dimensional distribution will be output, that is, the probability prediction values ​​corresponding to the K image categories respectively, and the first classification network and the second classification network will each obtain the output of the K-dimensional probability distribution. Then, for each image category, the average of the probability prediction value obtained in the first classification network and the probability prediction value obtained in the second classification network is calculated respectively, so as to obtain the target probability prediction value corresponding to each image category, and finally the image category with the largest target probability prediction value among these image categories is determined as the final classification result of the image to be classified.

[0037] A second aspect of an embodiment of the present application provides an image classification device, including:

[0038] An image acquisition module, used to acquire images to be classified;

[0039] A first image classification module, used for inputting the image to be classified into a first classification network, obtaining a first category prediction result of the image to be classified and a saliency map of the image to be classified, wherein the saliency map includes saliency values ​​of various positions in the image to be classified;

[0040] an image deformation processing module, configured to perform image deformation processing on the image to be classified according to the saliency map, so as to obtain a deformed image in which a target position in the image to be classified is enlarged, wherein the target position is a position in the image to be classified whose saliency value is greater than a specified value;

[0041] A second image classification module, used for inputting the deformed image into a second classification network to obtain a second category prediction result of the image to be classified;

[0042] A classification result determination module is used to determine the classification result of the image to be classified according to the first category prediction result and the second category prediction result.

[0043] A third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the image classification method provided in the first aspect of the embodiment of the present application are implemented.

[0044] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the image classification method provided in the first aspect of the embodiment of the present application are implemented.

[0045] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of the image classification method described in the first aspect of the embodiments of the present application.

[0046] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 is a flowchart of an embodiment of an image classification method provided by an embodiment of the present application;

[0049] Figure 2 is a flowchart of another embodiment of an image classification method provided by an embodiment of the present application;

[0050] Figure 3It is a schematic diagram of the operation of the image classification method proposed in the embodiment of the present application in an actual application scenario;

[0051] Figure 4 is a structural diagram of an embodiment of an image classification device provided in an embodiment of the present application;

[0052] Figure 5 It is a schematic diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures, technologies, etc. are proposed, so as to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed description of well-known systems, devices, circuits and methods is omitted to prevent unnecessary details from hindering the description of the present application. In addition, in the description of the present application specification and the attached claims, the terms "first", "second", "third" etc. are only used to distinguish the description, and cannot be interpreted as indicating or suggesting relative importance.

[0054] In view of the low accuracy of capsule endoscope image classification in the prior art, the embodiments of the present application propose an image classification method, apparatus, terminal device and storage medium, which can improve the accuracy of image classification. It should be understood that the execution subject of each method embodiment of the present application can be various types of terminal devices or servers, such as mobile phones, tablet computers, laptops, desktop computers, wearable devices and various medical image analysis devices.

[0055] See also Figure 1 , shows an image classification method proposed in an embodiment of the present application, including:

[0056] 101. Obtain an image to be classified;

[0057] First, an image to be classified is obtained. The image to be classified can be any type and theme of images that need to be classified. For example, the image to be classified can be a capsule endoscopy image of the inside of a human organ, which needs to be classified to determine whether it is a normal image or a pathological image.

[0058] 102. Input the image to be classified into a first classification network to obtain a first category prediction result of the image to be classified and a saliency map of the image to be classified;

[0059] After the image to be classified is obtained, it is input into a pre-trained first classification network to obtain a preliminary classification result for the image to be classified and a saliency map of the image to be classified. Specifically, the first classification network can be a neural network model of various structures, which is obtained by training a sample image set with category labels. After an image is input into the first classification network, the image features are extracted through the convolution layer of the network, and then the image features are processed by the fully connected layer to obtain the category prediction result of the image, that is, the probability distribution of the image belonging to each category in the preset multiple categories.

[0060] On the other hand, by inputting the image to be classified into the first classification network, a saliency map (also called attention map) of the image to be classified can also be obtained. A saliency map is an image that shows the uniqueness of each image pixel. It is a part of the image partition and is usually used to locate objects and boundaries in the image. Image saliency is an important visual feature in the image, reflecting the degree of attention paid by the human eye to each area of ​​the image. Specifically, the saliency map may include saliency values ​​(also called attention values) of each position in the image to be classified, which is used to mark which position in the image is more significant. The higher the saliency value, the higher the saliency of the corresponding image position in the overall image. The use of a saliency map has a great effect on the classification results of the image. For example, in a lesion image, the lesion area is marked because it has a larger saliency value. In the subsequent image classification process, the weight of the image features of the lesion area can be increased, thereby improving the accuracy of lesion image recognition.

[0061] In one embodiment of the present application, the first classification network has a plurality of convolution modules, and inputting the image to be classified into the first classification network to obtain a saliency map of the image to be classified may include:

[0062] (1) after the image to be classified is input into the first classification network, image features of the image to be classified are extracted by the multiple convolution modules;

[0063] (2) constructing a saliency map of the image to be classified according to the feature values ​​corresponding to the image features at various positions of the image to be classified.

[0064] The convolution module can be used to extract image features, and according to the characteristic values ​​of the image features at various positions in the image, the saliency values ​​at various positions in the image can be determined, thereby obtaining the saliency map of the image. For example, after a 224*224 image to be classified is input into the first network, it is extracted through multiple dense convolution modules (such as the feature image size of the last layer is 7*7*c, c is the number of channels of the feature image), and then the category probability distribution is predicted through the fully connected layer. The prediction result is supervised by the category label and optimized by the cross entropy loss function. After the first network is trained, the feature image of size 7*7*c will produce a higher response in the category-related area (such as the human tissue lesion area in the lesion image), that is, it has a higher image characteristic value, and the corresponding image saliency value is also higher. By averaging each image channel, the saliency map (attention map) of the image to be classified can be obtained.

[0065] 103. Performing image deformation processing on the image to be classified according to the saliency map to obtain a deformed image in which a target position in the image to be classified is enlarged;

[0066] After obtaining the saliency map of the image to be classified, the image to be classified is deformed according to the saliency map, thereby obtaining a deformed image that enlarges the target position in the image to be classified, wherein the target position is a position in the image to be classified where the saliency value is greater than a specified value. The saliency value of each position in the image can be known according to the saliency map, so that the target position with a larger saliency value in the image can be enlarged to obtain a deformed image. For the specific image deformation processing method, reference can be made to the next embodiment of the present application.

[0067] 104. Input the deformed image into a second classification network to obtain a second category prediction result of the image to be classified;

[0068] After obtaining the deformed image, the deformed image is input into a pre-trained second classification network to obtain a classification result for the deformed image. Specifically, the structure and type of the second classification network can be the same as or different from the first classification network. After a deformed image is input into the second classification network, a feature map of the deformed image is extracted through multiple densely connected convolution blocks, and then the feature map is processed by the fully connected layer to obtain the category prediction result of the image to be deformed, which can be regarded as the category prediction result of the image to be classified.

[0069] 105. Determine a classification result of the image to be classified according to the first category prediction result and the second category prediction result.

[0070] Finally, the category prediction results of the two classification networks for the image to be classified are combined to obtain the final classification result of the image to be classified. For example, the final classification result can be obtained by calculating the average value of the two category prediction results.

[0071] In one embodiment of the present application, the image classification method may further include:

[0072] Calculating the loss function of the second classification network according to the second category prediction result;

[0073] Optimizing the parameters of the first classification network according to the loss function of the second classification network.

[0074] In an embodiment of the present application, the two classification networks are interconnected, and the saliency map obtained by the first classification network can enlarge the target area in the image to be classified, thereby improving the accuracy of the second classification network for image recognition. Moreover, the parameters of the two classification networks can be trained and optimized, and can be trained in an end-to-end manner. If the saliency map generated by the first classification network is inaccurate, the image input to the second classification network will not achieve the desired deformation, resulting in low recognition accuracy of the second classification network and a large loss function. The loss function will generate a gradient back propagation to the first classification network to correct and optimize the parameters of the first classification network. In general, after obtaining the second category prediction result, the loss function of the second classification network can be calculated based on the second category prediction result, and then the gradient back propagation method is used to correct and optimize the parameters of the first classification network in combination with the loss function of the second classification network, thereby improving the performance of the first classification network.

[0075] In one embodiment of the present application, the image classification method may further include:

[0076] Calculating a loss function of the first classification network according to the first category prediction result;

[0077] Optimizing parameters of the second classification network according to the loss function of the first classification network.

[0078] Since the two classification networks are interconnected, the loss function of the first classification network can also be used to train and optimize the parameters of the second classification network. After obtaining the prediction results of the first category, the loss function of the first classification network can be calculated based on the prediction results of the first category, and then the parameters of the second classification network can be corrected and optimized by using the gradient back propagation method in combination with the loss function of the first classification network, thereby improving the performance of the second classification network.

[0079] In one embodiment of the present application, the first category prediction result includes a probability prediction value of each image category in a plurality of preset image categories, and the second category prediction result also includes a probability prediction value of each image category in the plurality of image categories. Determining the classification result of the image to be classified according to the first category prediction result and the second category prediction result may include:

[0080] (1) for each image category in the multiple image categories, respectively calculating an average of a probability prediction value of the image category in the first category prediction result and a probability prediction value of the image category in the second category prediction result, to obtain a target probability prediction value of each image category in the multiple image categories;

[0081] (2) Determine the image category with the largest target probability prediction value among the multiple image categories as the classification result of the image to be classified.

[0082] After the image is input into the first classification network, the image feature map will be extracted through multiple densely connected convolution blocks, and then the image feature map will be processed by the fully connected layer to obtain the probability prediction value of each category in the preset multiple image categories. The second classification network also outputs the probability prediction value of each category in the multiple image categories. For example, if there are K image categories in total, a K-dimensional distribution will be output, that is, the probability prediction values ​​corresponding to the K image categories respectively, and the first classification network and the second classification network each obtain the output of the K-dimensional probability distribution. Then, for each image category, the average of the probability prediction value obtained in the first classification network and the probability prediction value obtained in the second classification network is calculated respectively, so as to obtain the target probability prediction value corresponding to each image category, and finally the image category with the largest target probability prediction value among these image categories is determined as the final classification result of the image to be classified.

[0083] After acquiring the image to be classified, the embodiment of the present application inputs the image to be classified into a classification network to obtain a corresponding saliency image and a category prediction result. The saliency image marks the saliency value of each position in the image to be classified. Then, the image to be classified is deformed according to the saliency map to obtain a deformed image that enlarges the target position in the image. Next, another classification network is used to identify the deformed image to obtain another category prediction result, and finally the two category prediction results are combined to obtain the final image classification result. Taking the recognition of lesion images as an example, since the saliency value of the lesion area in the lesion image is large, a deformed image that enlarges the lesion area in the image will be obtained, and by enlarging the lesion area, the lesion features in the image will also correspondingly make a greater contribution to the features of the entire image, so the accuracy of lesion image recognition can be improved.

[0084] See also Figure 2 , shows another image classification method proposed in an embodiment of the present application, including:

[0085] 201. Obtain an image to be classified;

[0086] 202. Input the image to be classified into a first classification network to obtain a first category prediction result of the image to be classified and a saliency map of the image to be classified;

[0087] Steps 201-202 are the same as steps 101-102, and the details may refer to the relevant description of steps 101-102.

[0088] 203. Constructing a sampling point distribution image of the image to be classified according to the saliency map;

[0089] After obtaining the saliency map of the image to be classified, the sampling point distribution image of the image to be classified can be constructed according to the saliency map. The sampling point distribution image records the various sampling points distributed in the image to be classified, and the sampling point distribution density of each position in the image to be classified is proportional to the saliency value of the position. The sampling point distribution image can also be called a deformed sampling field, in which the larger the saliency value, the denser the sampling points are distributed, and the smaller the saliency value, the sparser the sampling points are distributed. The sampling point distribution image is essentially some irregularly distributed sampling points. For example, since the original size of the capsule endoscope image is 576*576 or 360*360, in order to reduce the computational complexity of the classification network, it is necessary to downsample the image before sending it to the classification network. For ordinary downsampling, it is achieved by uniformly sampling the original image. However, when using the deformed sampling field, the original image will be irregularly sampled during the downsampling process. For example, more sampling points will be allocated in the lesion area, and the sampled image will enlarge the lesion area.

[0090] Furthermore, constructing a sampling point distribution image of the image to be classified according to the saliency map may include:

[0091] (1) generating an initial image with uniformly distributed sampling points according to the size of the image to be classified;

[0092] (2) Based on the saliency values ​​of each position in the image to be classified contained in the saliency map, a convolution kernel of a Gaussian distribution is used to perform convolution processing on the initial image to obtain the sampling point distribution image.

[0093] When constructing the sampling point distribution image based on the saliency map, first, an initial image with uniformly distributed sampling points can be generated according to the size of the image to be classified, that is, the initial sampling field; then, the convolution kernel of the Gaussian distribution is used to convolve the initial sampling field with uniform distribution, in which each sampling point will pull the surrounding sampling points with a force proportional to its saliency value, so the target area with high saliency value will attract the surrounding sampling points with a greater force, thereby generating densely distributed sampling points. Conversely, the area with low saliency value has a relatively small pulling force on the surrounding sampling points, so the generated sampling points are sparsely distributed.

[0094] 204. Sampling the image to be classified according to the sampling point distribution image to obtain a deformed image that enlarges the target position in the image to be classified;

[0095] After obtaining the irregularly distributed sampling point distribution image, the image to be classified is sampled according to the sampling point distribution image, and a deformed image that enlarges the target position in the image to be classified can be obtained. Specifically, when deforming the image to be classified, the sampling density of the position is determined according to the saliency value of different positions in the image. The target area with a larger saliency value has a denser sampling; conversely, the general area with a smaller saliency value has a very sparse sampling. The image obtained by this sampling field deformation will enlarge the target area and compress the general area. For example, for a lesion image, the deformation of the sampling field will enlarge the lesion area and compress the normal non-lesion area.

[0096] 205. Input the deformed image into a second classification network to obtain a second category prediction result of the image to be classified;

[0097] 206. Determine a classification result of the image to be classified according to the first category prediction result and the second category prediction result.

[0098] Steps 205-206 are the same as steps 104-105, and the details may refer to the relevant description of steps 104-105.

[0099] After acquiring the image to be classified, the embodiment of the present application inputs the image to be classified into a classification network to obtain a corresponding saliency image and a category prediction result. The saliency image marks the saliency value of each position in the image to be classified. Then, a corresponding sampling point distribution image is constructed according to the saliency map, and the image to be classified is sampled based on the sampling point distribution image to obtain a deformed image that enlarges the target position in the image to be classified. Next, another classification network is used to identify the deformed image to obtain another category prediction result, and finally the two category prediction results are combined to obtain the final image classification result. Compared with the first embodiment of the present application, this embodiment proposes a specific implementation method for obtaining a deformed image corresponding to the image to be classified according to the saliency map.

[0100] For ease of understanding, the image classification method proposed in this application is explained below using a practical application scenario.

[0101] like Figure 3 , which is a schematic diagram of the operation of the image classification method proposed in an embodiment of the present application in an actual application scenario.

[0102] exist Figure 3 In , the image to be classified is an original capsule endoscopy image, and it is necessary to identify whether the image is a lesion image. The classification network 1 and the classification network 2 used are both DenseNet networks with the same structure. First, the capsule endoscopy image is input into the trained classification network 1. Under the supervision of the image label, the classification network 1 will output the category to which the capsule endoscopy image belongs (recorded as the diagnosis result 1), and automatically learn the attention map (saliency map). In this attention map, the attention value of the lesion area of ​​the input image is large, while the attention value of the non-lesion area of ​​the background is small. It can be seen that the attention map can achieve the role of lesion position annotation to a certain extent, indicating the approximate position of the lesion area in the image in an automatic learning manner. Then, according to the distribution of the attention values ​​of each image position point on the attention map, a deformed sampling field is constructed, in which the sampling points corresponding to the lesion area with a larger attention value are denser, and vice versa, the sampling points corresponding to the non-lesion area with a smaller attention value are sparser. Then, the deformed sampling field is used to sample the original capsule endoscopy image, and a deformed image that enlarges the lesion area in the image and compresses the non-lesion area can be obtained. After that, the deformed image is input into the classification network 2. Since the lesion area is magnified, the lesion features will also make a greater contribution to the features of the entire image accordingly. Therefore, the classification network 2 can obtain a more accurate image classification result, thereby improving the accuracy of the lesion image diagnosis result. The image classification result obtained by the classification network 2 is recorded as the diagnosis result 2. Finally, the final diagnosis result (final classification result) of the capsule endoscopy image can be obtained by combining the diagnosis result 1 and the diagnosis result 2. Specifically, the final diagnosis result can be obtained by calculating the average value of the diagnosis result 1 and the diagnosis result 2.

[0103] In addition, during the training phase of the classification network, both networks are supervised and optimized by the loss function calculated by the image label and the predicted value, which can promote each other. Specifically, if the attention map of the classification network 1 can accurately mark the area of ​​the lesion, the lesion area can be magnified, thereby improving the accuracy of the classification network 2. Finally, by integrating the two networks, the advantages of the two networks can be combined to obtain more accurate lesion image diagnosis results.

[0104] Based on the above content, it can be known that the present application can be used for disease diagnosis of capsule endoscopy images, and is particularly suitable for the recognition of lesion images with small disease areas, such as the recognition of lesion images of small areas of bleeding, ulcers and polyps in human tissue. It should be noted that the present application does not limit the category of the image to be classified, and can be applied to the detection and classification of any type of small objects under image labels.

[0105] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0106] An image classification method is mainly described above, and an image classification device will be described below.

[0107] See also Figure 4 , an embodiment of an image classification device in the embodiment of the present application includes:

[0108] An image acquisition module 401 is used to acquire an image to be classified;

[0109] A first image classification module 402, configured to input the image to be classified into a first classification network, and obtain a first category prediction result of the image to be classified and a saliency map of the image to be classified, wherein the saliency map includes saliency values ​​of various positions in the image to be classified;

[0110] An image deformation processing module 403 is used to perform image deformation processing on the image to be classified according to the saliency map to obtain a deformed image in which a target position in the image to be classified is enlarged, wherein the target position is a position in the image to be classified where the saliency value is greater than a specified value;

[0111] A second image classification module 404, configured to input the deformed image into a second classification network to obtain a second category prediction result of the image to be classified;

[0112] The classification result determination module 405 is used to determine the classification result of the image to be classified according to the first category prediction result and the second category prediction result.

[0113] In one embodiment of the present application, the first image classification module may include:

[0114] An image feature extraction unit, configured to extract image features of the image to be classified through the multiple convolution modules after the image to be classified is input into the first classification network;

[0115] The saliency map construction unit is used to construct a saliency map of the image to be classified according to the feature values ​​corresponding to the image features at various positions of the image to be classified.

[0116] In one embodiment of the present application, the image deformation processing module may include:

[0117] a sampling point distribution image construction unit, configured to construct a sampling point distribution image of the image to be classified according to the saliency map, wherein the sampling point distribution image records each sampling point distributed in the image to be classified, and the sampling point distribution density at each position in the image to be classified is proportional to the saliency value of the position;

[0118] An image sampling unit is used to sample the image to be classified according to the sampling point distribution image to obtain the deformed image.

[0119] Furthermore, the sampling point distribution image construction unit may include:

[0120] An initial image generating subunit, used for generating an initial image with uniformly distributed sampling points according to the size of the image to be classified;

[0121] The convolution processing subunit is used to perform convolution processing on the initial image using a convolution kernel of Gaussian distribution based on the saliency value of each position in the image to be classified contained in the saliency map, so as to obtain the sampling point distribution image.

[0122] In one embodiment of the present application, the image classification device may further include:

[0123] A second loss function calculation module, used to calculate the loss function of the second classification network according to the second category prediction result;

[0124] The first classification network optimization module is used to optimize the parameters of the first classification network according to the loss function of the second classification network.

[0125] In one embodiment of the present application, the image classification device may further include:

[0126] A first loss function calculation module, used to calculate the loss function of the first classification network according to the first category prediction result;

[0127] The second classification network optimization module is used to optimize the parameters of the second classification network according to the loss function of the first classification network.

[0128] In one embodiment of the present application, the first category prediction result includes a probability prediction value of each image category in a preset plurality of image categories, the second category prediction result also includes a probability prediction value of each image category in the plurality of image categories, and the classification result determination module may include:

[0129] a probability prediction value calculation unit, configured to calculate, for each image category in the plurality of image categories, an average of a probability prediction value of the image category in the first category prediction result and a probability prediction value of the image category in the second category prediction result, to obtain a target probability prediction value of each image category in the plurality of image categories;

[0130] The classification result determination unit is used to determine the image category with the largest target probability prediction value among the multiple image categories as the classification result of the image to be classified.

[0131] The present application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, Figure 1 or Figure 2 Represents the steps of any image classification method.

[0132] The embodiment of the present application also provides a computer program product, when the computer program product is run on a terminal device, the terminal device executes the following Figure 1 or Figure 2 Represents the steps of any image classification method.

[0133] Figure 5 Schematic diagram of a terminal device provided by an embodiment of the present application. Figure 5 As shown, the terminal device 5 of this embodiment includes: a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, the steps in the above-mentioned embodiments of each image classification method are implemented, for example Figure 2 Alternatively, when the processor 50 executes the computer program 52, the functions of each module / unit in the above-mentioned device embodiments are realized, for example, Figure 4 Functions of modules 401 to 405 are shown.

[0134] The computer program 52 may be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 52 in the terminal device 5.

[0135] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0136] The memory 51 may be an internal storage unit of the terminal device 5, such as a hard disk or memory of the terminal device 5. The memory 51 may also be an external storage device of the terminal device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the terminal device 5. The memory 51 is used to store the computer program and other programs and data required by the terminal device. The memory 51 may also be used to temporarily store data that has been output or is to be output.

[0137] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0140] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0141] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. . Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0145] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image classification method, characterized in that: include: Get the image to be classified; Inputting the image to be classified into a first classification network to obtain a first category prediction result of the image to be classified and a saliency map of the image to be classified, wherein the saliency map includes saliency values ​​of various positions in the image to be classified; Performing image deformation processing on the image to be classified according to the saliency map to obtain a deformed image that enlarges a target position in the image to be classified, wherein the target position is a position in the image to be classified where the saliency value is greater than a specified value; Inputting the deformed image into a second classification network to obtain a second category prediction result of the image to be classified; Determining a classification result of the image to be classified according to the first category prediction result and the second category prediction result; The step of performing image deformation processing on the image to be classified according to the saliency map to obtain a deformed image that enlarges the target position in the image to be classified includes: Constructing a sampling point distribution image of the image to be classified according to the saliency map, wherein the sampling point distribution image records each sampling point distributed in the image to be classified, and the sampling point distribution density at each position in the image to be classified is proportional to the saliency value of the position; Sampling the image to be classified according to the sampling point distribution image to obtain the deformed image; Constructing a sampling point distribution image of the image to be classified according to the saliency map, comprising: Generate an initial image with uniformly distributed sampling points according to the size of the image to be classified; Based on the saliency values ​​of each position in the image to be classified contained in the saliency map, a convolution kernel of Gaussian distribution is used to perform convolution processing on the initial image to obtain the sampling point distribution image.

2. The image classification method according to claim 1, characterized in that: The first classification network has a plurality of convolution modules, and the image to be classified is input into the first classification network to obtain a saliency map of the image to be classified, including: After the image to be classified is input into the first classification network, image features of the image to be classified are extracted by the multiple convolution modules; A saliency map of the image to be classified is constructed according to feature values ​​corresponding to the image features at various positions of the image to be classified.

3. The image classification method according to claim 1, characterized in that: Also includes: Calculating the loss function of the second classification network according to the second category prediction result; Optimizing the parameters of the first classification network according to the loss function of the second classification network.

4. The image classification method according to claim 1, characterized in that: Also includes: Calculating a loss function of the first classification network according to the first category prediction result; Optimizing parameters of the second classification network according to the loss function of the first classification network.

5. The image classification method according to any one of claims 1 to 4, characterized in that: The first category prediction result includes a probability prediction value of each image category in a plurality of preset image categories, the second category prediction result also includes a probability prediction value of each image category in the plurality of image categories, and determining a classification result of the image to be classified according to the first category prediction result and the second category prediction result includes: For each image category in the multiple image categories, respectively calculate an average of a probability prediction value of the image category in the first category prediction result and a probability prediction value of the image category in the second category prediction result to obtain a target probability prediction value of each image category in the multiple image categories; The image category with the largest target probability prediction value among the multiple image categories is determined as the classification result of the image to be classified.

6. An image classification device, characterized in that: include: An image acquisition module, used to acquire images to be classified; A first image classification module, used for inputting the image to be classified into a first classification network, obtaining a first category prediction result of the image to be classified and a saliency map of the image to be classified, wherein the saliency map includes saliency values ​​of various positions in the image to be classified; an image deformation processing module, configured to perform image deformation processing on the image to be classified according to the saliency map, so as to obtain a deformed image in which a target position in the image to be classified is enlarged, wherein the target position is a position in the image to be classified whose saliency value is greater than a specified value; A second image classification module, used for inputting the deformed image into a second classification network to obtain a second category prediction result of the image to be classified; A classification result determination module, used to determine the classification result of the image to be classified according to the first category prediction result and the second category prediction result; Wherein, the image deformation processing module includes: a sampling point distribution image construction unit, configured to construct a sampling point distribution image of the image to be classified according to the saliency map, wherein the sampling point distribution image records each sampling point distributed in the image to be classified, and the sampling point distribution density at each position in the image to be classified is proportional to the saliency value of the position; An image sampling unit, configured to sample the image to be classified according to the sampling point distribution image to obtain the deformed image; The sampling point distribution image construction unit comprises: An initial image generating subunit, used for generating an initial image with uniformly distributed sampling points according to the size of the image to be classified; The convolution processing subunit is used to perform convolution processing on the initial image using a convolution kernel of Gaussian distribution based on the saliency value of each position in the image to be classified contained in the saliency map, so as to obtain the sampling point distribution image.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image classification method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image classification method according to any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Image classification method, apparatus, terminal device, and storage medium

    CN109522942A

  • Image classification method and device, computer device and storage medium

    CN110287836A

  • Image recognition method and device and training method and device of image recognition model

    CN111680698A

  • Image recognition method and device based on figure fine-grained features

    CN111860250A

  • Image multi-label classification method, system and device and readable storage medium

    CN111897985A