An image classification method, device, equipment and storage medium thereof
By constructing an image classification method that includes a pre-defined interpretable model, the problem of insufficient interpretability of medical image classification results is solved, and highly interpretable image classification results are achieved, thereby improving trustworthiness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2024-06-28
- Publication Date
- 2026-07-28
AI Technical Summary
Existing technologies lack a reliable and simple interpretable method for interpreting medical image classification results, resulting in low trust levels.
An image classification method is adopted, which constructs a first processing component containing a pre-defined interpretable model and a second processing component containing a pre-trained image classification model to reconstruct images and generate highly interpretable image classification results.
It improves the interpretability and confidence of image classification results, helps doctors quickly identify abnormal or diseased tissues, and enables patients to understand the diagnostic results.
Smart Images
Figure CN118710981B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology and is applied to image classification scenarios, particularly to an image classification method, apparatus, device, and storage medium thereof. Background Technology
[0002] In recent years, deep learning has been widely applied in the medical field. For example, in pathological image classification, deep learning algorithms are used to diagnose tumor types and assess the quality of tissue samples. By training neural networks to identify different cell types and structures in images, pathologists can make diagnoses and treatment plans more quickly. Through training on a large number of medical images, image classification models can learn how to distinguish between normal and abnormal tissues, organs, and lesions.
[0003] Although deep learning has been widely applied in the medical field, a significant portion of the population remains skeptical about its use in image classification and assisting doctors in case analysis, believing that machine / deep learning-assisted case analysis is inherently prone to error. Currently, there is a lack of a convincing and easily understandable interpretable method for explaining medical image classification results, leading to low trust in these results. Summary of the Invention
[0004] The purpose of this application is to provide an image classification method, apparatus, device and storage medium to solve the problem that the existing technology lacks a convincing and simple interpretable method for interpreting medical image classification results, which leads to low trust in medical image classification results.
[0005] To address the aforementioned technical problems, this application provides an image classification method, employing the following technical solution:
[0006] An image classification method includes the following steps:
[0007] Obtain a batch of images to be classified and construct a target image set;
[0008] The target image set is input into a preset first processing component to obtain an image processed by the first processing component, wherein the first processing component includes an input layer, a convolutional layer and an output layer of a preset interpretable model;
[0009] The target image set is input into a preset second processing component to obtain an image processed by the second processing component, wherein the second processing component includes a pre-trained image classification model and an interpretable representation layer of the interpretable model;
[0010] According to a preset image reconstruction mechanism, the images processed by the first processing component and the images processed by the second processing component are reconstructed to obtain a reconstructed image set;
[0011] The reconstructed image set is input into the image classification model to obtain the classification results of the batch of images to be classified.
[0012] Furthermore, before performing the step of inputting the target image set into a preset first processing component to obtain an image processed by the first processing component, the method further includes:
[0013] Obtain the training image set with labeled classification results as the target image set;
[0014] The target image set is input into the first processing component to be trained to obtain the image processed by the first processing component.
[0015] The target image set is input into a preset second processing component to obtain an image processed by the second processing component;
[0016] According to a preset image reconstruction mechanism, the images processed by the first processing component and the images processed by the second processing component are reconstructed to obtain a reconstructed image set;
[0017] The reconstructed image set is input into the image classification model to obtain the actual classification results of the training image set;
[0018] Based on the actual classification results of the training image set and the labeled classification results, the output value of the last softmax layer of the image classification model is obtained;
[0019] Based on the output value and the preset output threshold, the first processing component is tuned and trained until the output value meets the output threshold. Then the tuning and training of the first processing component is completed, and the trained first processing component is used as the preset first processing component.
[0020] Furthermore, the step of inputting the target image set into the first processing component to be trained to obtain the image processed by the first processing component specifically includes:
[0021] The training image set with labeled classification results is input into the convolutional layer via the input layer;
[0022] The convolutional layer performs convolutional blurring on all images in the training image set to obtain blurred images corresponding to all images in the training image set. The convolutional layer is composed of a single-layer convolutional neural network.
[0023] The output layer outputs blurred images corresponding to all images in the training image set.
[0024] Furthermore, the step of inputting the target image set into a preset first processing component to obtain an image processed by the first processing component specifically includes:
[0025] The batch of images to be classified are input into the convolutional layer via the input layer;
[0026] The batch of images to be classified are subjected to convolutional blurring processing through the convolutional layer to obtain blurred images corresponding to all images to be classified.
[0027] The output layer outputs blurred images corresponding to all images to be classified.
[0028] Furthermore, the step of inputting the target image set into a preset second processing component to obtain an image processed by the second processing component specifically includes:
[0029] The image classification model is used to extract features from all images in the target image set to obtain feature extraction results for each image. The feature extraction includes image edge feature extraction and / or image texture feature extraction.
[0030] Based on the feature extraction results corresponding to all images and the regional attention levels corresponding to different preset feature factors, the interpretation and representation layer generates heatmaps corresponding to all images in the target image set. The heatmaps use different colors and color intensities to represent the regional attention levels corresponding to different feature factors.
[0031] Furthermore, the step of reconstructing the image processed by the first processing component and the image processed by the second processing component according to a preset image reconstruction mechanism to obtain a reconstructed image set specifically includes:
[0032] The heatmap is subjected to image binarization to obtain a corresponding black and white image;
[0033] From the original image corresponding to the heatmap, extract the original image region corresponding to the white area of the black-and-white image, and
[0034] From the blurred image corresponding to the original image, extract the blurred image region corresponding to the black region of the black and white image;
[0035] A reconstructed image corresponding to the original image is generated by overlaying the original image region corresponding to the white region and the blurred image region corresponding to the black region, wherein the original image refers to any image in the current target image set;
[0036] Generate reconstructed images corresponding to all images in the current target image set to obtain the reconstructed image set.
[0037] Furthermore, the step of obtaining the output value of the last softmax layer of the image classification model based on the actual classification results of the training image set and the labeled classification results specifically includes:
[0038] The actual classification results of the training image set are compared with the labeled classification results, and the cross-entropy loss function is used to calculate the loss value after training the first processing component.
[0039] According to the preset difference calculation method:
[0040] O = 1 - L,
[0041] Calculate the output value, where O represents the output value and L represents the loss value, with L ranging from 0 to 1.
[0042] Furthermore, the step of optimizing and training the first processing component based on the output value and a preset output threshold until the output value meets the output threshold, thus completing the optimization and training of the first processing component, and using the trained first processing component as the preset first processing component, specifically includes:
[0043] Step 501: Determine whether the output value exceeds the output threshold;
[0044] Step 502: If the output value does not exceed the output threshold, then perform parameter tuning on the convolutional layer of the first processing component, and after the parameter tuning is completed, perform the training step on the first processing component again to obtain the output value of the last softmax layer of the image classification model, and repeat step 501.
[0045] Step 503: Once the output value exceeds the output threshold, the first processing component is tuned and trained, and the trained first processing component is used as the preset first processing component.
[0046] To address the aforementioned technical problems, embodiments of this application also provide an image classification device, which employs the following technical solution:
[0047] An image classification device, comprising:
[0048] The image acquisition module is used to acquire a batch of images to be classified and construct a target image set;
[0049] The first processing module is used to input the target image set into a preset first processing module to obtain an image processed by the first processing module. The first processing module includes an input layer, a convolutional layer, and an output layer of a preset interpretable model.
[0050] The second processing module is used to input the target image set into a preset second processing module to obtain an image processed by the second processing module. The second processing module includes a pre-trained image classification model and an interpretation representation layer of the interpretable model.
[0051] The image reconstruction module is used to reconstruct the images processed by the first processing component and the images processed by the second processing component according to a preset image reconstruction mechanism, so as to obtain a reconstructed image set;
[0052] The reconstructed image classification module is used to input the reconstructed image set into the image classification model to obtain the classification results of the batch of images to be classified.
[0053] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:
[0054] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the image classification method described above.
[0055] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:
[0056] A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the steps of the image classification method described above.
[0057] Compared with the prior art, the embodiments of this application have the following main advantages:
[0058] The image classification method described in this application involves: inputting the image to be classified into a preset first processing component to obtain an image processed by the first processing component; inputting the image to be classified into a preset second processing component to obtain an image processed by the second processing component; reconstructing the images processed by the first and second processing components to obtain a reconstructed image; and inputting the reconstructed image into an image classification model to obtain an image classification result. By introducing an interpretability model into the first and second processing components, the interpretability and confidence of the image classification result are improved. In particular, introducing the interpretability model into medical image classification scenarios can not only assist doctors in accurately and quickly identifying abnormal or diseased tissues or organs, but also enable patients to quickly understand the doctor's diagnosis based on the interpretable representation. Attached Figure Description
[0059] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0060] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;
[0061] Figure 2 This is a flowchart of an embodiment of the image classification method according to this application;
[0062] Figure 3 This is a flowchart of a specific embodiment of training the first processing component in the image classification method described in this application;
[0063] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 302 shown;
[0064] Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 307 shown;
[0065] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 202 shown;
[0066] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 203 shown;
[0067] Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 204 shown;
[0068] Figure 9 This is a schematic diagram of the structure of an embodiment of the image classification apparatus according to this application;
[0069] Figure 10 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation
[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.
[0071] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0072] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0073] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0074] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.
[0075] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.
[0076] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.
[0077] It should be noted that the image classification method provided in this application embodiment is generally executed by a terminal device, and correspondingly, the image classification device is generally set in the terminal device.
[0078] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0079] Continue to refer to Figure 2 A flowchart of an embodiment of the image classification method according to this application is shown. The image classification method includes the following steps:
[0080] Step 201: Obtain a batch of images to be classified and construct the target image set.
[0081] In this embodiment, the image to be classified includes medical images to be classified and identified by doctors. By training on a large number of medical images, a corresponding image classification model can be obtained. This model can learn how to distinguish between normal and abnormal tissues, organs, and lesions, and identify different cell types and structures in the images. This can help pathologists make diagnoses and treatment plans more quickly. In pathological image classification, it can predict tumor types and assess the quality of tissue samples. Commonly used medical images include cardiovascular angiography images, X-ray CT scan images, ultrasound CT images, magnetic resonance imaging images, and medical ultrasound examination images.
[0082] Step 202: Input the target image set into a preset first processing component to obtain an image processed by the first processing component, wherein the first processing component includes an input layer, a convolutional layer and an output layer of a preset interpretable model.
[0083] The input layer, convolutional layer, and output layer of the preset interpretable model in the first processing component are used to obtain a blurred image of the target image, that is, a blurred image of the medical image of the patient that has been taken and is to be classified and identified by the doctor, so as to facilitate the subsequent use of the blurred image.
[0084] In this embodiment, before performing the step of inputting the target image set into a preset first processing component to obtain an image processed by the first processing component, the method further includes a method for training the first processing component.
[0085] Continue to refer to Figure 3 , Figure 3 This is a flowchart of a specific embodiment of training the first processing component in the image classification method described in this application, including:
[0086] Step 301: Obtain the training image set with labeled classification results as the target image set;
[0087] In this embodiment, the training image set with labeled classification results includes medical images with labeled classification results, such as medical images labeled with normal and abnormal cell tissues, or medical images labeled with tumor types. By using a training image set with labeled classification results, the confidence level of the trained interpretable model is guaranteed.
[0088] Step 302: Input the target image set into the first processing component to be trained to obtain the image processed by the first processing component;
[0089] Continue to refer to Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 302 shown includes:
[0090] Step 401: Input the training image set with labeled classification results into the convolutional layer via the input layer;
[0091] Step 402: Perform convolutional blurring processing on all images in the training image set through the convolutional layer to obtain blurred images corresponding to all images in the training image set, wherein the convolutional layer is composed of a single-layer convolutional neural network;
[0092] Step 403: Output the blurred images corresponding to all images in the training image set through the output layer.
[0093] By sequentially passing labeled medical images (marked with normal and abnormal cells or tissues, or labeled with tumor types) through the output layer, convolutional layer, and output layer of the interpretable model, a corresponding blurred image is output. The convolutional layer can be constructed using a single-layer convolutional neural network with a kernel size of 9×9. This ensures that the steps for processing the medical images of patients already taken and awaiting classification by doctors to obtain blurred images are consistent with those during training, thus increasing the reliability of the interpretable model.
[0094] Step 303: Input the target image set into a preset second processing component to obtain the image processed by the second processing component;
[0095] In this embodiment, the step of inputting the target image set into a preset second processing component to obtain an image processed by the second processing component specifically includes: performing feature extraction on all images in the target image set using the image classification model to obtain feature extraction results corresponding to each image, wherein the feature extraction includes image edge feature extraction and / or image texture feature extraction; and generating heatmaps corresponding to all images in the target image set through the interpretation and representation layer based on the feature extraction results corresponding to each image and the regional attention levels corresponding to preset different feature factors, wherein the heatmap refers to using different colors and color intensities to represent the regional attention levels corresponding to different feature factors.
[0096] Specifically, the step of extracting features from all images in the target image set using the image classification model to obtain the feature extraction results for each image is implemented as follows: An edge detection algorithm is introduced into the feature extraction layer of the image classification model to extract image edge features. For example, an edge detection algorithm is introduced into the feature extraction layer of the image classification model based on a CNN convolutional neural network to extract the image edge features. Similarly, a texture descriptor factor can be introduced into the feature extraction layer of the image classification model to extract image texture features. For example, a texture descriptor factor is introduced into the feature extraction layer of the image classification model based on a CNN convolutional neural network to extract the image texture features.
[0097] The image classification model extracts features from medical images labeled with normal and abnormal cell tissues, or labeled with tumor types. Edge features and / or texture features are extracted from the medical images. Based on the interpretable model's interpretable representation layer, different colors and intensities are used to represent the regional attention levels corresponding to different feature factors, generating a heatmap corresponding to the medical image. This allows the interpretable model to directly indicate where the image classification model's focus is. During the interpretable model training phase, the confidence level of interpretability is guaranteed. The use of different colors and intensities in the heatmap makes it easy for doctors and patients to view and understand the content represented in the heatmap.
[0098] Step 304: According to a preset image reconstruction mechanism, perform image reconstruction on the image processed by the first processing component and the image processed by the second processing component to obtain a reconstructed image set;
[0099] In this embodiment, the step of reconstructing the images processed by the first processing component and the images processed by the second processing component according to a preset image reconstruction mechanism to obtain a reconstructed image set specifically includes: performing image binarization processing on the heatmap to obtain a corresponding black-and-white image; extracting the original image region corresponding to the white area of the black-and-white image from the original image corresponding to the heatmap, and extracting the blurred image region corresponding to the black area of the black-and-white image from the blurred image corresponding to the original image; generating a reconstructed image corresponding to the original image by overlaying the original image region corresponding to the white area and the blurred image region corresponding to the black area, wherein the original image refers to any image in the current target image set; generating reconstructed images corresponding to all images in the current target image set respectively to obtain the reconstructed image set. At this time, the current target image set refers to the training image set with labeled classification results.
[0100] In this embodiment, extracting the original image region corresponding to the white area of the black-and-white image from the original image corresponding to the heatmap, and extracting the blurred image region corresponding to the black area of the black-and-white image from the blurred image corresponding to the original image, can be replaced by extracting the heatmap region corresponding to the white area of the black-and-white image from the heatmap, and extracting the blurred image region corresponding to the black area of the black-and-white image from the blurred image corresponding to the original image; generating a reconstructed image corresponding to the original image by overlaying the heatmap region corresponding to the white area and the blurred image region corresponding to the black area, wherein the original image refers to any image in the current target image set; generating reconstructed images corresponding to all images in the current target image set respectively, obtaining the reconstructed image set. At this time, the current target image set refers to the training image set with labeled classification results.
[0101] Specifically, the step of performing image binarization processing on the heatmap to obtain the corresponding black and white image can be achieved by performing a dilation operation on the heatmap so that the size of the heatmap is similar to the size of the blurred image. Then, only the red channel image of the RGB three colors corresponding to the heatmap is extracted. By calculating the mean of all pixels in the red channel image, pixels with a value less than the mean are uniformly converted to white, and pixels with a value greater than the mean are uniformly converted to black, thereby realizing the image binarization processing of the heatmap to obtain the corresponding black and white image.
[0102] In this embodiment, when performing the steps of extracting the original image region corresponding to the white area of the black-and-white image from the original image corresponding to the heatmap, and extracting the blurred image region corresponding to the black area of the black-and-white image from the blurred image corresponding to the original image, the optimal solution is to first perform a dilation operation to make the heatmap, the blurred image, and the original image the same size before extracting the original image region and the blurred image region, so as to facilitate subsequent image reconstruction.
[0103] In essence, the image reconstruction process involves hiding and blurring the regions corresponding to features with low attention, while enhancing or not enhancing the features with high attention, and then reconstructing the image. This selective enhancement or reduction of features in image regions allows doctors or patients to focus more directly on key feature regions without wasting a lot of time on irrelevant or less influential feature regions.
[0104] Step 305: Input the reconstructed image set into the image classification model to obtain the actual classification result of the training image set;
[0105] Step 306: Based on the actual classification results of the training image set and the labeled classification results, obtain the output value of the last softmax layer of the image classification model;
[0106] In this embodiment, the step of obtaining the output value of the last softmax layer of the image classification model based on the actual classification result of the training image set and the labeled classification result specifically includes: comparing the actual classification result of the training image set and the labeled classification result, and using the cross-entropy loss function to calculate the loss value after training the first processing component; calculating the output value according to the preset difference calculation method: O = 1 - L, where O represents the output value, L represents the loss value, and the value of L is in the range of 0 to 1.
[0107] Step 307: Based on the output value and the preset output threshold, the first processing component is tuned and trained until the output value meets the output threshold. Then the tuning and training of the first processing component is completed, and the trained first processing component is used as the preset first processing component.
[0108] Continue to refer to Figure 5 , Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 307 shown includes:
[0109] Step 501: Determine whether the output value exceeds the output threshold;
[0110] Step 502: If the output value does not exceed the output threshold, then perform parameter tuning on the convolutional layer of the first processing component, and after the parameter tuning is completed, perform the training step on the first processing component again to obtain the output value of the last softmax layer of the image classification model, and repeat step 501.
[0111] Specifically, since the image classification model is already trained, there is no need for extensive model tuning. Moreover, the interpretable model consists of an input layer, a convolutional layer, an interpretation representation layer, and an output layer. Only the convolutional layer needs to be continuously tuned, without incurring significant training costs, to achieve interpretable output of image classification results.
[0112] Step 503: Once the output value exceeds the output threshold, the first processing component is tuned and trained, and the trained first processing component is used as the preset first processing component.
[0113] Continue to refer to Figure 6 , Figure 6 yes Figure 2A flowchart of a specific embodiment of step 202 shown includes:
[0114] Step 601: Input the batch of images to be classified into the convolutional layer via the input layer;
[0115] Step 602: Perform convolutional blurring processing on the batch of images to be classified through the convolutional layer to obtain blurred images corresponding to all images to be classified.
[0116] Step 603: Output the blurred images corresponding to all the images to be classified through the output layer.
[0117] In this embodiment, the image to be classified is a medical image that has just been taken of the patient and is to be classified and identified by the doctor. The medical image is input into a first processing component that has been trained. Since the medical image needs to be interpreted and represented to obtain a heat map, the medical image can also be input into an interpretable model that has been trained. The first processing component or the interpretable model can then be used to obtain a fuzzy map corresponding to the medical image.
[0118] Step 203: Input the target image set into a preset second processing component to obtain the image processed by the second processing component, wherein the second processing component includes a pre-trained image classification model and an interpretation representation layer of the interpretable model.
[0119] Continue to refer to Figure 7 , Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 203 shown includes:
[0120] Step 701: Perform feature extraction on all images in the target image set using the image classification model to obtain feature extraction results for each image. The feature extraction includes image edge feature extraction and / or image texture feature extraction.
[0121] Correspondingly, for medical images of patients that have just been taken and are about to be classified and identified by doctors, feature extraction can be performed by introducing an edge detection algorithm into the feature extraction layer of the image classification model to extract image edge features. For example, an edge detection algorithm can be introduced into the feature extraction layer of the image classification model based on a CNN convolutional neural network to extract image edge features. Similarly, texture descriptors can be introduced into the feature extraction layer of the image classification model to extract image texture features. For example, texture descriptors can be introduced into the feature extraction layer of the image classification model based on a CNN convolutional neural network to extract image texture features. In practical applications, it is necessary to ensure that the edge detection algorithm is used during training and is also used in subsequent actual classification prediction, or that texture descriptors are used during training and are also used in subsequent actual classification prediction, so as to ensure consistency in feature extraction.
[0122] Step 702: Based on the feature extraction results corresponding to all images and the regional attention levels corresponding to different preset feature factors, the interpretation and representation layer generates heatmaps corresponding to all images in the target image set. The heatmap refers to the regional attention levels corresponding to different feature factors represented by different colors and color intensities.
[0123] The image classification model extracts features from newly taken medical images of patients, which are then classified and identified by doctors. Edge and / or texture features are extracted from the images. Based on the interpretable model's interpretive representation layer, different colors and intensities are used to represent the regional attention levels corresponding to different feature factors, generating a heatmap of the medical image. This allows the interpretable model to directly indicate the specific focus of the image classification model, improving image classification accuracy by identifying key classification locations, thus playing a more significant auxiliary role in medical applications.
[0124] Step 204: According to the preset image reconstruction mechanism, perform image reconstruction on the image processed by the first processing component and the image processed by the second processing component to obtain a reconstructed image set.
[0125] Continue to refer to Figure 8 , Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 204 shown includes:
[0126] Step 801: Perform image binarization processing on the heat map to obtain the corresponding black and white image;
[0127] In this embodiment, to simplify the processing steps, the method of performing image binarization processing in step 801 to obtain the corresponding black and white image is preferably consistent with the method of performing image binarization processing to obtain the corresponding black and white image in the aforementioned training steps.
[0128] Step 802: Extract the original image region corresponding to the white area of the black-and-white image from the original image corresponding to the heatmap, and
[0129] Step 803: Extract the blurred image region corresponding to the black region of the black and white image from the blurred image corresponding to the original image;
[0130] Step 804: By superimposing the original image region corresponding to the white region and the blurred image region corresponding to the black region, a reconstructed image corresponding to the original image is generated, wherein the original image refers to any image in the current target image set;
[0131] Step 805: Generate reconstructed images corresponding to all images in the current target image set, thus obtaining the reconstructed image set. At this point, the current target image set refers to the image set comprised of the batch of images to be classified.
[0132] In this embodiment, step 802, which involves extracting the original image region corresponding to the white area of the black-and-white image from the original image corresponding to the heatmap, can be replaced by extracting the heatmap image region corresponding to the white area of the black-and-white image from the heatmap. Correspondingly, step 804 involves generating the reconstructed image corresponding to the original image by overlaying the heatmap image region corresponding to the white area and the blurred image region corresponding to the black area. Here, the original image refers to any image in the current target image set. In this case, the current target image set refers to the image set composed of the batch of images to be classified.
[0133] In actual image classification, the system hides and blurs the regions corresponding to features with low attention, while enhancing or not enhancing the features with high attention. Then, the image is reconstructed. The system selectively enhances or weakens features in the image region, which allows doctors or patients to focus more on key feature regions without wasting a lot of time on irrelevant or less influential feature regions.
[0134] Step 205: Input the reconstructed image set into the image classification model to obtain the classification results of the batch of images to be classified.
[0135] Of course, in the actual classification prediction stage, the input image to be classified can also be a single image. Whether it is a single image to be classified or a batch of images to be classified depends on the actual application scenario and does not affect the processing steps proposed in this application.
[0136] This application obtains an image to be classified by inputting it into a preset first processing component to obtain an image processed by the first processing component; inputting the image to be classified into a preset second processing component to obtain an image processed by the second processing component; reconstructing the images processed by the first and second processing components to obtain a reconstructed image; and inputting the reconstructed image into an image classification model to obtain an image classification result. By introducing an interpretability model into the first and second processing components, the interpretability and confidence of the image classification result are improved. In particular, introducing the interpretability model into the medical image classification scenario can not only assist doctors in accurately and quickly identifying abnormal or diseased tissues or organs, but also enable patients to quickly understand the doctor's diagnosis based on the interpretable representation.
[0137] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0138] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0139] In this embodiment, an image to be classified is input into a preset first processing component to obtain an image processed by the first processing component; the image to be classified is then input into a preset second processing component to obtain an image processed by the second processing component; the images processed by the first and second processing components are reconstructed to obtain a reconstructed image; and the reconstructed image is input into an image classification model to obtain an image classification result. By introducing an interpretability model into the first and second processing components, the interpretability and confidence of the image classification result are improved. In particular, introducing the interpretability model into the medical image classification scenario can not only assist doctors in accurately and quickly identifying abnormal or diseased tissues or organs, but also enable patients to quickly understand the doctor's diagnosis based on the interpretable representation.
[0140] Further reference Figure 9 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of an image classification device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0141] like Figure 9 As shown, the image classification device 900 described in this embodiment includes: an image acquisition module 901, a first processing component processing module 902, a second processing component processing module 903, an image reconstruction module 904, and a reconstructed image classification module 905. Wherein:
[0142] The image acquisition module 901 is used to acquire a batch of images to be classified and construct a target image set.
[0143] The first processing module 902 is used to input the target image set into a preset first processing module to obtain an image processed by the first processing module. The first processing module includes an input layer, a convolutional layer and an output layer of a preset interpretable model.
[0144] The second processing module 903 is used to input the target image set into a preset second processing module to obtain an image processed by the second processing module. The second processing module includes a pre-trained image classification model and an interpretation representation layer of the interpretable model.
[0145] Image reconstruction module 904 is used to reconstruct the image processed by the first processing component and the image processed by the second processing component according to a preset image reconstruction mechanism to obtain a reconstructed image set;
[0146] The reconstructed image classification module 905 is used to input the reconstructed image set into the image classification model to obtain the classification results of the batch of images to be classified.
[0147] This application obtains an image to be classified by inputting it into a preset first processing component to obtain an image processed by the first processing component; inputting the image to be classified into a preset second processing component to obtain an image processed by the second processing component; reconstructing the images processed by the first and second processing components to obtain a reconstructed image; and inputting the reconstructed image into an image classification model to obtain an image classification result. By introducing an interpretability model into the first and second processing components, the interpretability and confidence of the image classification result are improved. In particular, introducing the interpretability model into the medical image classification scenario can not only assist doctors in accurately and quickly identifying abnormal or diseased tissues or organs, but also enable patients to quickly understand the doctor's diagnosis based on the interpretable representation.
[0148] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).
[0149] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0150] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.
[0151] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c that are interconnected via a system bus. It should be noted that only the computer device 10 with components 10a-10c is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital digital processors (DSPs), embedded devices, etc.
[0152] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.
[0153] The memory 10a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a may be an internal storage unit of the computer device 10, such as the hard disk or memory of the computer device 10. In other embodiments, the memory 10a may also be an external storage device of the computer device 10, such as a plug-in hard disk, smart memory card (SMC), secure digital (SD) card, flash memory card, etc. Of course, the memory 10a may also include both the internal storage unit and the external storage device of the computer device 10. In this embodiment, the memory 10a is typically used to store the operating system and various application software installed on the computer device 10, such as computer-readable instructions for an image classification method. Furthermore, the memory 10a can also be used to temporarily store various types of data that have been output or will be output.
[0154] In some embodiments, the processor 10b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 10b is typically used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to execute computer-readable instructions stored in the memory 10a or to process data, for example, to execute computer-readable instructions for the image classification method.
[0155] The network interface 10c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 10 and other electronic devices.
[0156] The computer device proposed in this embodiment belongs to the field of artificial intelligence technology and is applied in image classification scenarios. This application obtains an image to be classified by inputting it into a preset first processing component to obtain an image processed by the first processing component; inputting the image to be classified into a preset second processing component to obtain an image processed by the second processing component; reconstructing the images processed by the first and second processing components to obtain a reconstructed image; and inputting the reconstructed image into an image classification model to obtain an image classification result. By introducing an interpretability model into the first and second processing components, the interpretability and confidence of the image classification results are improved. In particular, introducing the interpretability model into medical image classification scenarios can assist doctors in accurately and quickly identifying abnormal or diseased tissues or organs, and also enable patients to quickly understand the doctor's diagnosis based on the interpreted representation.
[0157] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the image classification method described above.
[0158] The computer-readable storage medium proposed in this embodiment belongs to the field of artificial intelligence technology and is applied in image classification scenarios. This application obtains an image to be classified by inputting it into a preset first processing component to obtain an image processed by the first processing component; inputting the image to be classified into a preset second processing component to obtain an image processed by the second processing component; reconstructing the images processed by the first and second processing components to obtain a reconstructed image; and inputting the reconstructed image into an image classification model to obtain an image classification result. By introducing an interpretability model into the first and second processing components, the interpretability and confidence of the image classification results are improved. In particular, introducing the interpretability model into medical image classification scenarios can assist doctors in accurately and quickly identifying abnormal or diseased tissues or organs, and also enable patients to quickly understand the doctor's diagnosis based on the interpreted representation.
[0159] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0160] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.
Claims
1. An image classification method, characterized in that, Includes the following steps: Obtain a batch of images to be classified and construct a target image set; The target image set is input into a preset first processing component to obtain an image processed by the first processing component. The first processing component includes an input layer, a convolutional layer, and an output layer of a preset interpretable model, specifically including: The target image set is input into the convolutional layer via the input layer; The convolutional layer performs convolutional blurring on all images in the target image set to obtain blurred images corresponding to all images in the target image set. The convolutional layer is composed of a single-layer convolutional neural network. The output layer outputs blurred images corresponding to all images in the target image set. The target image set is input into a preset second processing component to obtain images processed by the second processing component. The second processing component includes a pre-trained image classification model and an interpretable representation layer of the interpretable model, specifically including: The image classification model is used to extract features from all images in the target image set to obtain feature extraction results for each image. The feature extraction includes image edge feature extraction and / or image texture feature extraction. Based on the feature extraction results corresponding to all images and the regional attention levels corresponding to different preset feature factors, the interpretation and representation layer generates heatmaps corresponding to all images in the target image set. The heatmaps refer to the regional attention levels corresponding to different feature factors represented by different colors and color intensities. According to a preset image reconstruction mechanism, image reconstruction is performed on the image processed by the first processing component and the image processed by the second processing component to obtain a reconstructed image set, specifically including: The heatmap is binarized to obtain a corresponding black-and-white image. Specifically, the heatmap is dilated to make its size match that of the blurred image; only the red channel image from the RGB color depth of the heatmap is extracted; the mean value of all pixels in the red channel image is calculated; by comparison, pixels smaller than the mean value are uniformly converted to white, and pixels larger than the mean value are uniformly converted to black; thus, the black-and-white image is obtained. From the original image corresponding to the heatmap, extract the original image region corresponding to the white area of the black-and-white image, and From the blurred image corresponding to the original image, extract the blurred image region corresponding to the black region of the black and white image; A reconstructed image corresponding to the original image is generated by overlaying the original image region corresponding to the white region and the blurred image region corresponding to the black region, wherein the original image refers to any image in the current target image set; Generate reconstructed images corresponding to all images in the current target image set to obtain the reconstructed image set; The reconstructed image set is input into the image classification model to obtain the classification results of the batch of images to be classified.
2. The image classification method according to claim 1, characterized in that, Before performing the step of inputting the target image set into a preset first processing component to obtain an image processed by the first processing component, the method further includes: Obtain the training image set with labeled classification results as the target image set; The target image set is input into the first processing component to be trained to obtain the image processed by the first processing component. The target image set is input into a preset second processing component to obtain an image processed by the second processing component; According to a preset image reconstruction mechanism, the images processed by the first processing component and the images processed by the second processing component are reconstructed to obtain a reconstructed image set; The reconstructed image set is input into the image classification model to obtain the actual classification results of the training image set; Based on the actual classification results of the training image set and the labeled classification results, the output value of the last softmax layer of the image classification model is obtained; Based on the output value and the preset output threshold, the first processing component is tuned and trained until the output value meets the output threshold. Then the tuning and training of the first processing component is completed, and the trained first processing component is used as the preset first processing component.
3. The image classification method according to claim 2, characterized in that, The step of inputting the target image set into the first processing component to be trained, and obtaining the image processed by the first processing component, specifically includes: The training image set with labeled classification results is input into the convolutional layer via the input layer; The convolutional layer performs convolutional blurring on all images in the training image set to obtain blurred images corresponding to all images in the training image set. The convolutional layer is composed of a single-layer convolutional neural network. The output layer outputs blurred images corresponding to all images in the training image set.
4. The image classification method according to claim 2, characterized in that, The step of obtaining the output value of the last softmax layer of the image classification model based on the actual classification results of the training image set and the labeled classification results specifically includes: The actual classification results of the training image set are compared with the labeled classification results, and the cross-entropy loss function is used to calculate the loss value after training the first processing component. According to the preset difference calculation method: ， Calculate the output value, where, This represents the output value. This represents the loss value. The value of is between 0 and 1.
5. The image classification method according to claim 2, characterized in that, The step of optimizing and training the first processing component based on the output value and a preset output threshold until the output value meets the output threshold, and then using the trained first processing component as the preset first processing component, specifically includes: Step 501: Determine whether the output value exceeds the output threshold; Step 502: If the output value does not exceed the output threshold, then perform parameter tuning on the convolutional layer of the first processing component, and after the parameter tuning is completed, perform the training step on the first processing component again to obtain the output value of the last softmax layer of the image classification model, and repeat step 501. Step 503: Once the output value exceeds the output threshold, the first processing component is tuned and trained, and the trained first processing component is used as the preset first processing component.
6. An image classification device, characterized in that, The image classification device implements the steps of the image classification method as described in any one of claims 1 to 5, and the image classification device comprises: The image acquisition module is used to acquire a batch of images to be classified and construct a target image set; The first processing module is used to input the target image set into a preset first processing module to obtain an image processed by the first processing module. The first processing module includes an input layer, a convolutional layer, and an output layer of a preset interpretable model. The second processing module is used to input the target image set into a preset second processing module to obtain an image processed by the second processing module. The second processing module includes a pre-trained image classification model and an interpretation representation layer of the interpretable model. The image reconstruction module is used to reconstruct the images processed by the first processing component and the images processed by the second processing component according to a preset image reconstruction mechanism, so as to obtain a reconstructed image set; The reconstructed image classification module is used to input the reconstructed image set into the image classification model to obtain the classification results of the batch of images to be classified.
7. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the image classification method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the image classification method as described in any one of claims 1 to 5.