Image processing method and device, computer device and medium

Abstract

This application relates to the field of digital medical technology, and in particular to an image processing method, apparatus, computer equipment, and medium. The method involves acquiring an image to be predicted; preprocessing the image to be predicted to obtain a binary image; performing data augmentation on the binary image to obtain an augmented first image set; inputting the augmented image into a pre-trained deep learning model to obtain a prediction result set; and performing attribution processing on the prediction result set to obtain an attribution map set corresponding to the prediction result set. This application considers the reliability differences between different prediction results for the same pathological image, assigning different weight values ​​to the prediction results using reliability scores, so that different prediction results have different degrees of importance in the merging process. This solution improves the reliability of the prediction results for pathological images.

Description

Technical Field

[0001] This application relates to the field of digital medical technology, and in particular to an image processing method, apparatus, computer equipment, and medium. Background Technology

[0002] Currently, diagnosis based on pathological images primarily relies on manual screening by specialized physicians. With the increasing number of patients and the ever-growing demands for diagnostic accuracy, the number of pathological images requiring analysis has multiplied, necessitating more personnel and equipment to handle the growing volume of histopathological analysis needs. However, given my country's current situation, experienced physicians are scarce and their skill levels vary widely. The digitization level of pathology departments is generally low, and digital pathology equipment is severely lacking, posing significant challenges to the further development of pathological diagnosis. Considering the convenience of storage and remote transmission, digital histopathological images are receiving increasing attention. More importantly, digital histopathological images offer the possibility of introducing intelligent auxiliary analysis to reduce the burden on physicians, which is of great significance in alleviating the strain on medical resources.

[0003] In the field of computer vision (CV), due to the uncertainty of deep learning models, the inference results of a single input image may not be accurate. In order to improve the prediction performance of deep learning models, without changing the model's structure and parameters, multiple transformations can be applied to the input image, and inference can be performed on both the original and transformed images simultaneously to obtain multiple prediction results. These prediction results are then merged to obtain the final prediction result. This method of augmenting data during inference is called test-time augmentation (TTA).

[0004] In test-time augmentation, methods for merging multiple prediction results include taking the average, geometric mean, and maximum value of the prediction results. These merging methods have the following drawbacks: First, they do not consider the differences in the reliability of the prediction results. Some prediction results with very poor reliability may affect the accuracy of the merged result, causing the TTA result to fail to fully utilize the information of multiple prediction results after augmentation. Second, the prediction results of each sample have the same weight, ignoring the advantages of prediction results with high reliability. Summary of the Invention

[0005] The main objective of this application is to provide an image processing method, apparatus, computer device, and storage medium that can solve the technical problem that the prediction results obtained when using deep learning models for image prediction in the prior art are unreliable.

[0006] This application provides an image processing method, including:

[0007] Obtain the image to be predicted;

[0008] The image to be predicted is preprocessed to obtain a binary image;

[0009] The binary image is augmented to obtain the augmented first image set;

[0010] After data augmentation, the image to be predicted is input into a pre-trained deep learning model to obtain a set of prediction results.

[0011] The prediction result set is subjected to attribution processing to obtain an attribution map set corresponding to the prediction result set;

[0012] The final prediction result is obtained by performing calculations based on the first image set and the attribution map set.

[0013] Furthermore, the step of preprocessing the image to be predicted to obtain a binary image includes:

[0014] Obtain a pre-trained segmentation model, wherein the segmentation model includes at least a segmentation module, the segmentation module includes multiple convolutional network blocks and at least one convolutional layer, the multiple convolutional network blocks are connected sequentially and then connected to the at least one convolutional layer, and each convolutional network block includes a convolutional layer, a batch normalization layer and a non-linear activation layer.

[0015] The image to be predicted is input into the segmentation model, and the segmentation model outputs a binary image corresponding to the first image.

[0016] Furthermore, the step of preprocessing the image to be predicted to obtain a binary image includes:

[0017] The image to be predicted is converted to grayscale to obtain a grayscale image;

[0018] The grayscale image is processed according to a preset first pixel threshold to obtain an intermediate binary image;

[0019] Using the intermediate binary image as a guide image, the grayscale image is subjected to guided filtering to obtain a filtered image;

[0020] Based on a preset second pixel threshold, high-value pixels in the filtered image are determined, wherein the grayscale value of the high-value pixels is greater than the second pixel threshold;

[0021] The grayscale values ​​of the high-value pixels are augmented according to a preset augmentation coefficient to obtain an augmented image.

[0022] The augmented image is then sharpened to obtain a clear image;

[0023] The contrast of the clear image is adjusted to obtain the binary image.

[0024] Furthermore, the step of performing data augmentation on the binary image to obtain the augmented first image set includes:

[0025] The binary image is augmented by one or more of the following methods: flipping, scaling, translating, dithering, mirroring, or cropping, to obtain an augmented first image set.

[0026] Furthermore, the step of calculating the final prediction result based on the first image set and the attribution graph set includes:

[0027] Obtain a first image from the first image set, and obtain an attribution map corresponding to the first image from the attribution map set;

[0028] The reliability score corresponding to the attribution map is obtained by multiplying the pixels of the first image and the pixels of the attribution map point by point and then dividing by the sum of the pixel values ​​of the attribution map.

[0029] Obtain the reliability scores corresponding to all attribution maps in the attribution map set;

[0030] Input the reliability scores corresponding to all attribution maps into the softmax function and scale them to obtain the weights corresponding to all prediction results in the prediction result set.

[0031] Based on the weights, all prediction results in the prediction result set are linearly weighted and summed to obtain the final prediction result.

[0032] Further, the step of inputting the reliability scores corresponding to all attribution maps into the softmax function and scaling them to obtain the weights corresponding to all prediction results in the prediction result set includes:

[0033] Formula used:

[0034] w=softmax([R0, R1, R2,..., R N-1 The weights corresponding to all prediction results in the prediction result set are obtained by ] / T), where w is the weight, T is the temperature factor, and T is adjusted according to the specific application scenario. N-1 The reliability score is N, and N-1 is the index of the prediction result.

[0035] This application also provides an image processing apparatus, comprising:

[0036] The image acquisition module is used to acquire the image to be predicted.

[0037] The binary image acquisition module is used to preprocess the image to be predicted to obtain a binary image;

[0038] The first image set acquisition module is used to perform data augmentation on the binary image to obtain an enhanced first image set.

[0039] The prediction result set acquisition module is used to perform data augmentation on the image to be predicted and then input it into a pre-trained deep learning model to obtain the prediction result set.

[0040] The attribution graph acquisition module is used to perform attribution processing on the prediction result set to obtain the attribution graph set corresponding to the prediction result set.

[0041] The prediction result acquisition module is used to perform calculations based on the first image set and the attribution map set to obtain the final prediction result.

[0042] Furthermore, the prediction result acquisition module includes:

[0043] The first image and attribution map acquisition submodule is used to acquire a first image from the first image set and acquire an attribution map corresponding to the first image from the attribution map set.

[0044] The calculation submodule is used to multiply the pixels of the first image and the pixels of the attribution map point by point, and then divide by the sum of the pixel values ​​of the attribution map to obtain the reliability score corresponding to the attribution map.

[0045] The reliability score acquisition submodule is used to acquire the reliability scores corresponding to all attribution maps in the attribution map set.

[0046] The weight acquisition submodule is used to input the reliability scores corresponding to all attribution maps into the softmax function and scale them to obtain the weights corresponding to all prediction results in the prediction result set.

[0047] The prediction result acquisition submodule is used to perform a linear weighted summation of all prediction results in the prediction result set based on the weights to obtain the final prediction result.

[0048] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0049] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0050] Compared with existing technologies, this application provides an image processing method, apparatus, computer equipment, and medium, which acquires an image to be predicted; preprocesses the image to be predicted to obtain a binary image; performs data augmentation on the binary image to obtain an augmented first image set; after data augmentation, the image to be predicted is input into a pre-trained deep learning model to obtain a prediction result set; and performs attribution processing on the prediction result set to obtain an attribution map set corresponding to the prediction result set. This application considers the reliability differences between different prediction results for the same sample, and assigns different weight values ​​to the prediction results using reliability scores, so that different prediction results have different degrees of importance in the merging process. This scheme improves the reliability of the prediction results. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the steps of an image processing method in one embodiment of this application;

[0052] Figure 2 This is a schematic diagram of the steps of an image processing method in another embodiment of this application;

[0053] Figure 3 This is a structural block diagram of an image processing apparatus according to an embodiment of this application;

[0054] Figure 4 This is a schematic block diagram of an embodiment of the computer device of this application;

[0055] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0058] Reference Figure 1 This application provides an image processing method, comprising:

[0059] S1. Obtain the image to be predicted;

[0060] S2. Preprocess the image to be predicted to obtain a binary image;

[0061] S3. Perform data augmentation on the binary image to obtain the augmented first image set;

[0062] S4. After performing data augmentation on the image to be predicted, input it into a pre-trained deep learning model to obtain a prediction result set.

[0063] S5. Perform attribution processing on the prediction result set to obtain an attribution graph set corresponding to the prediction result set;

[0064] S6. Calculate based on the first image set and the attribution map set to obtain the final prediction result.

[0065] Currently, diagnosis based on pathological images mainly relies on manual screening by professional doctors. With the increasing number of patients and the ever-increasing demands for diagnostic accuracy, the number of pathological images requiring analysis has multiplied, necessitating more personnel and equipment to handle the growing demand for histopathological analysis. In the field of Computer Vision (CV), due to the inherent uncertainty of deep learning models, the prediction results from a single input pathological image may be inaccurate. In steps S1-S6 above, the reliability of the prediction results for each pathological image is quantified, and the reliability score is used as a weight value when merging prediction results to improve the reliability of the final prediction result. This application is generally divided into two parts: reliability score calculation and prediction result merging. For the prediction results of the deep learning model, this application defines the reliability score of the prediction result as the magnitude of the contribution of the target region of the pathological image to the prediction result. The reliability score calculation relies on a visual attribution algorithm. To avoid modifying the model's structure and parameters, this scheme uses a post-hoc visual attribution algorithm. Compared with existing technologies, this application has the following advantages: First, this scheme is interpretable. First, this scheme utilizes a visual attribution algorithm to generate an attribution graph during the merging of prediction results, allowing for interpretation of the predictions and providing a decision-making basis for visualizing the deep learning model. Second, it considers the reliability differences between different predictions for the same sample, assigning different weights to each prediction result based on its reliability score, thus giving different prediction results varying degrees of importance during the merging process. Third, this scheme improves the reliability of the prediction results. The test-time enhancement based on reliability quantification proposed in this scheme assigns very small weights to prediction results with low reliability scores during the merging process, thereby eliminating the adverse effects of unreliable predictions on the final result and obtaining more reliable predictions.

[0066] In one feasible implementation, step S2, which preprocesses the image to be predicted to obtain a binary image, includes:

[0067] S201. Obtain a pre-trained segmentation model, wherein the segmentation model includes at least a segmentation module, the segmentation module includes multiple convolutional network blocks and at least one convolutional layer, the multiple convolutional network blocks are connected in sequence and then connected to the at least one convolutional layer, and each convolutional network block includes a convolutional layer, a batch normalization layer and a non-linear activation layer.

[0068] S202. Input the image to be predicted into the segmentation model, and the segmentation model outputs the binary image corresponding to the first image.

[0069] In steps S201 and S202 above, image segmentation is a fundamental topic in the field of computer vision. Image segmentation is the technique and process of dividing an image into several specific regions with unique properties and extracting the target of interest. It is a key step from image processing to image analysis. Users can pre-train a segmentation model according to their needs, so that the segmentation model can output the segmentation mask required by the user. For example, if the user needs to use the trained model to perform portrait segmentation, then the segmentation model obtained after training should be a model that can output a portrait segmentation mask. Similarly, if the user needs to use the trained model to segment a specific object (such as a car or a potted plant), then the segmentation model obtained after training should be a model that can output a segmentation mask for that specific object, and so on. The image to be predicted is input into the segmentation model, and the segmentation model outputs the segmentation mask corresponding to the image to be predicted. For example, after acquiring the image to be predicted and the segmentation model, the electronic device can input the image to be predicted into the segmentation model, and the segmentation model will output the segmentation mask corresponding to the image to be predicted. The segmentation mask described above is a binary image. A pixel value of 1 indicates that the pixel belongs to the target region, and a pixel value of 0 indicates that the pixel belongs to a non-target region.

[0070] In one feasible implementation, the segmentation model further includes a multi-scale encoder module, a feature pyramid module, a multi-scale decoder module, and a deep feature supervision module. The multi-scale encoder module, the feature pyramid module, the multi-scale decoder module, and the segmentation module are connected in sequence. The deep feature supervision module and the feature pyramid module are used to supervise deep features from multiple scales.

[0071] In this embodiment, the model, comprising four modules—a multi-scale encoder module, a feature pyramid module, a multi-scale decoder module, and a deep feature supervision module—can be trained first; this training is referred to as the first stage of training. The input image is input through the multi-scale encoder module, processed by it, and then transmitted to the feature pyramid module. After processing by the feature pyramid module, it is transmitted to the deep feature supervision module and the multi-scale decoder module, respectively. The image data, after processing by the deep feature supervision module, yields masks upsampled at four scales, for example, denoted as Mask32, Mask16, Mask8, and Mask4. The multi-scale decoder module outputs the first preliminary segmentation mask. The base network in the multi-scale encoder can be the MobileNetV2 network, which has strong feature extraction capabilities and is relatively lightweight, and then feature maps at different scales are extracted to form a feature pyramid. In the feature pyramid module, the numbers 320, 64, 32, and 24 on the feature pyramid image represent the number of channels, while the numbers 1 / 4, 1 / 8, 1 / 16, and 1 / 32 represent the multiples of the resolution after downsampling relative to the original image.

[0072] In one feasible implementation, step S2, which preprocesses the image to be predicted to obtain a binary image, includes:

[0073] S211. Perform grayscale processing on the image to be predicted to obtain a grayscale image;

[0074] S212. The grayscale image is processed according to a preset first pixel threshold to obtain an intermediate binary image.

[0075] S213. Using the intermediate binary image as a guide image, perform guided filtering on the grayscale image to obtain a filtered image;

[0076] S214. Based on a preset second pixel threshold, determine high-value pixels in the filtered image, wherein the grayscale value of the high-value pixels is greater than the second pixel threshold.

[0077] S215. Based on a preset expansion coefficient, the grayscale value of the high-value pixel is expanded to obtain an expanded image.

[0078] S216. The expanded image is sharpened to obtain a clear image;

[0079] S217. Adjust the contrast of the clear image to obtain the binary image.

[0080] In step S211 above, the grayscale processing methods include component method, maximum value method, average value method, and weighted average method.

[0081] In step S212 above, a thresholding method can be used to binarize the grayscale image to obtain an intermediate binarized image. Commonly used binarization threshold selection methods include the bimodal method, the P-parameter method, the Otsu's method, the maximum entropy method, and the iterative method. The first pixel threshold can be selected using any of these methods. The first pixel threshold can be set according to the actual situation and is not specifically limited here.

[0082] In step S213 above, in the guided filtering process, the intermediate binarized image is used as the guide image, the grayscale image is used as the input image in the guided filtering process, and the filtered image is used as the output image in the guided filtering process. Thus, by performing guided filtering on the grayscale image through the intermediate binarized image, a filtered image that is generally similar to the grayscale image and whose edge texture is similar to the intermediate binarized image can be output. After guided filtering, the noise in the image is significantly reduced.

[0083] In step S214 above, the second pixel threshold is the sum of the mean gray value and the standard deviation of the gray value of the filtered image. That is, the second pixel threshold is equal to the average gray value of each pixel in the filtered image plus the standard deviation of the gray value of each pixel in the filtered image.

[0084] In step S215 above, the preset expansion coefficient is 1.2 to 1.5, for example, 1.3. The gray value of each high-value pixel is multiplied by the preset expansion coefficient to expand the gray value of the high-value pixels, thereby obtaining an expanded image with more obvious black and white contrast.

[0085] In step S216 above, the expanded image is sharpened to obtain a sharp image. This includes: applying Gaussian filtering to blur the expanded image to obtain a blurred image corresponding to the expanded image; and mixing the blurred image corresponding to the expanded image and the expanded image proportionally according to a preset mixing coefficient to obtain a sharp image. By sharpening the expanded image, a sharper image than the expanded image can be obtained.

[0086] In step S217 above, adjusting the contrast of the clear image includes: adjusting the grayscale value of each pixel in the clear image based on the average grayscale value of the clear image. Therefore, by adjusting the contrast of the clear image, a binary image with more pronounced black-and-white contrast can be obtained.

[0087] Further, step S3, which involves performing data augmentation on the binary image to obtain the augmented first image set, includes:

[0088] The binary image is augmented by one or more of the following methods: flipping, scaling, translating, dithering, mirroring, or cropping, to obtain an augmented first image set.

[0089] Digital histopathological imaging offers the possibility of introducing intelligent assisted analysis to reduce the burden on doctors, which is of great significance in alleviating the strain on medical resources. Current mainstream intelligent assisted analysis methods include image feature extraction, deep learning models, etc., but these methods often require a large number of data samples. Taking deep learning as an example, firstly, there needs to be as many training samples as possible, and secondly, the distribution of samples needs to be uniform. However, in reality, there are often situations where the sample size is insufficient, failing to meet the training requirements of deep learning models. By performing data augmentation on the binary image through one or more of the methods mentioned in step S3 above—flipping, scaling, translation, shaking, mirroring, or cutting—expanded samples can be formed, effectively expanding the original pathological samples. This solves the problems of insufficient sample quantity and uneven sample distribution to a certain extent, meeting the large sample size requirements of deep learning models. It effectively avoids overfitting and insufficient generalization ability in the trained model, improving the reliability of the assisted analysis results. The core of this step lies in the image preprocessing stage before training the deep learning model after random image transformation, effectively solving the problems of excessively large pathological slice images and insufficient sample size. In this application, the binary image obtained from the histopathological image is denoted as M. After enhancing the binary image, an expanded sample set is obtained, namely the first image set {M0, M1, M2, M3, ..., M}. N-1}, where M0 = M.

[0090] Further, step S6, which calculates the final prediction result based on the first image set and the attribution graph set, includes:

[0091] S61. Obtain a first image from the first image set, and obtain an attribution map corresponding to the first image from the attribution map set;

[0092] S62. Multiply the pixels of the first image and the pixels of the attribution map point by point, and then divide by the sum of the pixel values ​​of the attribution map to obtain the reliability score corresponding to the attribution map.

[0093] S63. Obtain the reliability scores corresponding to all attribution maps in the attribution map set;

[0094] S64. Input the reliability scores corresponding to all attribution maps into the softmax function and scale them to obtain the weights corresponding to all prediction results in the prediction result set.

[0095] S65. Based on the weights, perform a linear weighted summation of all prediction results in the prediction result set to obtain the final prediction result.

[0096] Specifically, in step S4, let the obtained prediction result set be {Y0, Y1, Y2, ..., Y}.N-1}, then the attribution graph corresponding to the predicted result set is {L0, L1, L2, ..., L}. N-1 In this embodiment, a post-hoc visual attribution algorithm (e.g., Grad-CAM) is used to attribute the N prediction results to obtain the corresponding attribution maps. L i With the corresponding M i Pixel-by-pixel multiplication and division by L i The reliability score is obtained by summing the pixel values. Refer to the following formula:

[0097]

[0098] Where i is the index of the prediction result, Sum() is the summation function, and R... i This represents the reliability score.

[0099] Prediction results with high reliability scores will be given greater importance in the merging process. The steps for merging prediction results are as follows:

[0100] The reliability scores {R0, R1, R2, ..., R...} of the N prediction results are used to determine the reliability scores of the prediction results. N-1 Input the softmax function and apply temperature scaling to obtain the weights of N prediction results.

[0101] w=softmax([R0, R1, R2,..., R N-1 ] / T)

[0102] Where w is the weight and T is the temperature factor, which is adjusted according to the specific application scenario.

[0103] The final prediction result is obtained by linearly weighting and summing the N prediction results using the following formula, where o is the final prediction result.

[0104]

[0105] Through steps S61-S64 above, the reliability differences between different prediction results for the same pathological sample are considered. Different weight values ​​are assigned to the prediction results using reliability scores, giving different prediction results varying degrees of importance during the merging process. This improves the reliability of the prediction results, thus eliminating the adverse effects of unreliable predictions on the final result and yielding more reliable predictions.

[0106] Reference Figure 3 This application also provides an image processing apparatus, comprising:

[0107] Image acquisition module 1 is used to acquire the image to be predicted;

[0108] Binary image acquisition module 2 is used to preprocess the image to be predicted to obtain a binary image;

[0109] The first image set acquisition module 3 is used to perform data augmentation on the binary image to obtain an enhanced first image set;

[0110] The prediction result set acquisition module 4 is used to perform data augmentation on the image to be predicted and then input it into a pre-trained deep learning model to obtain the prediction result set.

[0111] Attribution graph acquisition module 5 is used to perform attribution processing on the prediction result set to obtain an attribution graph set corresponding to the prediction result set.

[0112] The prediction result acquisition module 6 is used to perform calculations based on the first image set and the attribution map set to obtain the final prediction result.

[0113] Furthermore, the first image set acquisition module 3 includes:

[0114] A segmentation model acquisition submodule is used to acquire a pre-trained segmentation model, wherein the segmentation model includes at least a segmentation module, the segmentation module includes multiple convolutional network blocks and at least one convolutional layer, the multiple convolutional network blocks are connected in sequence and then connected to the at least one convolutional layer, and each convolutional network block includes a convolutional layer, a batch normalization layer and a non-linear activation layer.

[0115] The binary image acquisition submodule is used to input the image to be predicted into the segmentation model, and the segmentation model outputs the binary image corresponding to the first image.

[0116] Furthermore, the binary image acquisition module 2 includes:

[0117] The grayscale image acquisition submodule is used to perform grayscale processing on the image to be predicted to obtain a grayscale image;

[0118] The intermediate binary image acquisition submodule is used to process the grayscale image according to a preset first pixel threshold to obtain an intermediate binary image.

[0119] The filtered image acquisition submodule is used to perform guided filtering on the grayscale image using the intermediate binary image as a guide image to obtain a filtered image.

[0120] The high-value pixel acquisition submodule is used to determine high-value pixels in the filtered image according to a preset second pixel threshold, wherein the gray value of the high-value pixel is greater than the second pixel threshold;

[0121] An expanded image acquisition submodule is used to expand the gray values ​​of the high-value pixels according to a preset expansion coefficient to obtain an expanded image.

[0122] The clear image acquisition submodule is used to perform sharpening processing on the augmented image to obtain a clear image;

[0123] The binary image acquisition submodule is used to adjust the contrast of the clear image to obtain the binary image.

[0124] Furthermore, the first image set acquisition module 3 includes:

[0125] The binary image is augmented by one or more of the following methods: flipping, scaling, translating, dithering, mirroring, or cropping, to obtain an augmented first image set.

[0126] Furthermore, the prediction result acquisition module includes:

[0127] The first image and attribution map acquisition submodule is used to acquire a first image from the first image set and acquire an attribution map corresponding to the first image from the attribution map set.

[0128] The calculation submodule is used to multiply the pixels of the first image and the pixels of the attribution map point by point, and then divide by the sum of the pixel values ​​of the attribution map to obtain the reliability score corresponding to the attribution map.

[0129] The reliability score acquisition submodule is used to acquire the reliability scores corresponding to all attribution maps in the attribution map set.

[0130] The weight acquisition submodule is used to input the reliability scores corresponding to all attribution maps into the softmax function and scale them to obtain the weights corresponding to all prediction results in the prediction result set.

[0131] The prediction result acquisition submodule is used to perform a linear weighted summation of all prediction results in the prediction result set based on the weights to obtain the final prediction result.

[0132] Furthermore, the weight acquisition submodule uses the following formula:

[0133] w=softmax([R0, R1, R2,..., R N-1 The weights corresponding to all prediction results in the prediction result set are obtained by ] / T), where w is the weight, T is the temperature factor, and T is adjusted according to the specific application scenario. N-1 The reliability score is N, and N-1 is the index of the prediction result.

[0134] Reference Figure 4 This application also provides a computer device whose internal structure can be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, display device, and input device connected via a system bus. The network interface is used for communication with external terminals via a network connection. The display device is used to display interactive pages. The input device is used to receive user input. The processor is designed to provide computing and control capabilities. The memory includes non-volatile storage media. This non-volatile storage media stores the operating system, computer programs, and a database. The database stores raw data. When executed by the processor, the computer program implements an image processing method.

[0135] The processor described above executes the image processing method described above to obtain an image to be predicted; preprocesses the image to be predicted to obtain a binary image; performs data augmentation on the binary image to obtain an augmented first image set; after data augmentation, the image to be predicted is input into a pre-trained deep learning model to obtain a prediction result set; the prediction result set is subjected to attribution processing to obtain an attribution map set corresponding to the prediction result set; this application considers the reliability differences between different prediction results of the same sample, and uses reliability scores to assign different weight values ​​to the prediction results, so that different prediction results have different degrees of importance in the merging process. This scheme improves the reliability of the prediction results.

[0136] Furthermore, the step of preprocessing the image to be predicted to obtain a binary image includes:

[0137] Obtain a pre-trained segmentation model, wherein the segmentation model includes at least a segmentation module, the segmentation module includes multiple convolutional network blocks and at least one convolutional layer, the multiple convolutional network blocks are connected sequentially and then connected to the at least one convolutional layer, and each convolutional network block includes a convolutional layer, a batch normalization layer and a non-linear activation layer.

[0138] The image to be predicted is input into the segmentation model, and the segmentation model outputs a binary image corresponding to the first image.

[0139] Furthermore, the step of preprocessing the image to be predicted to obtain a binary image includes:

[0140] The image to be predicted is converted to grayscale to obtain a grayscale image;

[0141] The grayscale image is processed according to a preset first pixel threshold to obtain an intermediate binary image;

[0142] Using the intermediate binary image as a guide image, the grayscale image is subjected to guided filtering to obtain a filtered image;

[0143] Based on a preset second pixel threshold, high-value pixels in the filtered image are determined, wherein the grayscale value of the high-value pixels is greater than the second pixel threshold;

[0144] The grayscale values ​​of the high-value pixels are augmented according to a preset augmentation coefficient to obtain an augmented image.

[0145] The augmented image is then sharpened to obtain a clear image;

[0146] The contrast of the clear image is adjusted to obtain the binary image.

[0147] Furthermore, the step of performing data augmentation on the binary image to obtain the augmented first image set includes:

[0148] The binary image is augmented by one or more of the following methods: flipping, scaling, translating, dithering, mirroring, or cropping, to obtain an augmented first image set.

[0149] Furthermore, the step of calculating the final prediction result based on the first image set and the attribution graph set includes:

[0150] Obtain a first image from the first image set, and obtain an attribution map corresponding to the first image from the attribution map set;

[0151] The reliability score corresponding to the attribution map is obtained by multiplying the pixels of the first image and the pixels of the attribution map point by point and then dividing by the sum of the pixel values ​​of the attribution map.

[0152] Obtain the reliability scores corresponding to all attribution maps in the attribution map set;

[0153] Input the reliability scores corresponding to all attribution maps into the softmax function and scale them to obtain the weights corresponding to all prediction results in the prediction result set.

[0154] Based on the weights, all prediction results in the prediction result set are linearly weighted and summed to obtain the final prediction result.

[0155] Further, the step of inputting the reliability scores corresponding to all attribution maps into the softmax function and scaling them to obtain the weights corresponding to all prediction results in the prediction result set includes:

[0156] Formula used:

[0157] w=softmax([R0, R1, R2,..., R N-1The weights corresponding to all prediction results in the prediction result set are obtained by ] / T), where w is the weight, T is the temperature factor, and T is adjusted according to the specific application scenario. N-1 The reliability score is N, and N-1 is the index of the prediction result.

[0158] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by the processor, the computer program implements an image processing method: acquiring an image to be predicted; preprocessing the image to be predicted to obtain a binary image; performing data augmentation on the binary image to obtain an augmented first image set; inputting the data-augmented image to be predicted into a pre-trained deep learning model to obtain a prediction result set; and performing attribution processing on the prediction result set to obtain an attribution map set corresponding to the prediction result set. This application considers the reliability differences between different prediction results for the same sample, and assigns different weight values ​​to the prediction results using reliability scores, so that different prediction results have different degrees of importance in the merging process. This scheme improves the reliability of the prediction results.

[0159] Furthermore, the step of preprocessing the image to be predicted to obtain a binary image includes:

[0160] Obtain a pre-trained segmentation model, wherein the segmentation model includes at least a segmentation module, the segmentation module includes multiple convolutional network blocks and at least one convolutional layer, the multiple convolutional network blocks are connected sequentially and then connected to the at least one convolutional layer, and each convolutional network block includes a convolutional layer, a batch normalization layer and a non-linear activation layer.

[0161] The image to be predicted is input into the segmentation model, and the segmentation model outputs a binary image corresponding to the first image.

[0162] Furthermore, the step of preprocessing the image to be predicted to obtain a binary image includes:

[0163] The image to be predicted is converted to grayscale to obtain a grayscale image;

[0164] The grayscale image is processed according to a preset first pixel threshold to obtain an intermediate binary image;

[0165] Using the intermediate binary image as a guide image, the grayscale image is subjected to guided filtering to obtain a filtered image;

[0166] Based on a preset second pixel threshold, high-value pixels in the filtered image are determined, wherein the grayscale value of the high-value pixels is greater than the second pixel threshold;

[0167] The grayscale values ​​of the high-value pixels are augmented according to a preset augmentation coefficient to obtain an augmented image.

[0168] The augmented image is then sharpened to obtain a clear image;

[0169] The contrast of the clear image is adjusted to obtain the binary image.

[0170] Furthermore, the step of performing data augmentation on the binary image to obtain the augmented first image set includes:

[0171] The binary image is augmented by one or more of the following methods: flipping, scaling, translating, dithering, mirroring, or cropping, to obtain an augmented first image set.

[0172] Furthermore, the step of calculating the final prediction result based on the first image set and the attribution graph set includes:

[0173] Obtain a first image from the first image set, and obtain an attribution map corresponding to the first image from the attribution map set;

[0174] The reliability score corresponding to the attribution map is obtained by multiplying the pixels of the first image and the pixels of the attribution map point by point and then dividing by the sum of the pixel values ​​of the attribution map.

[0175] Obtain the reliability scores corresponding to all attribution maps in the attribution map set;

[0176] Input the reliability scores corresponding to all attribution maps into the softmax function and scale them to obtain the weights corresponding to all prediction results in the prediction result set.

[0177] Based on the weights, all prediction results in the prediction result set are linearly weighted and summed to obtain the final prediction result.

[0178] Further, the step of inputting the reliability scores corresponding to all attribution maps into the softmax function and scaling them to obtain the weights corresponding to all prediction results in the prediction result set includes:

[0179] Formula used:

[0180] w=softmax([R0, R1, R2,..., R N-1 The weights corresponding to all prediction results in the prediction result set are obtained by ] / T), where w is the weight, T is the temperature factor, and T is adjusted according to the specific application scenario. N-1 The reliability score is N, and N-1 is the index of the prediction result.

[0181] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0182] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0183] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be predicted; The image to be predicted is preprocessed to obtain a binary image; The binary image is augmented to obtain the augmented first image set; Each enhanced image in the first image set is input into a pre-trained deep learning model to obtain a prediction result set. The prediction result set is subjected to attribution processing to obtain an attribution map set corresponding to the prediction result set; The final prediction result is obtained by calculating based on the first image set and the attribution map set; The step of performing data augmentation on the binary image to obtain the augmented first image set includes: The binary image is augmented by one or more of the following methods: flipping, scaling, translating, dithering, mirroring, or cropping, to obtain an augmented first image set. The step of calculating the final prediction result based on the first image set and the attribution map set includes: Obtain a first image from the first image set, and obtain an attribution map corresponding to the first image from the attribution map set; The reliability score corresponding to the attribution map is obtained by multiplying the pixels of the first image and the pixels of the attribution map point by point and then dividing by the sum of the pixel values ​​of the attribution map. Obtain the reliability scores corresponding to all attribution maps in the attribution map set; Input the reliability scores corresponding to all attribution maps into the softmax function and scale them to obtain the weights corresponding to all prediction results in the prediction result set. Based on the weights, all prediction results in the prediction result set are linearly weighted and summed to obtain the final prediction result.

2. The image processing method according to claim 1, characterized in that, The step of preprocessing the image to be predicted to obtain a binary image includes: Obtain a pre-trained segmentation model, wherein the segmentation model includes at least a segmentation module, the segmentation module includes multiple convolutional network blocks and at least one convolutional layer, the multiple convolutional network blocks are connected sequentially and then connected to the at least one convolutional layer, and each convolutional network block includes a convolutional layer, a batch normalization layer and a non-linear activation layer. The image to be predicted is input into the segmentation model, and the segmentation model outputs a binary image corresponding to the image to be predicted.

3. The image processing method according to claim 1, characterized in that, The step of preprocessing the image to be predicted to obtain a binary image includes: The image to be predicted is converted to grayscale to obtain a grayscale image; The grayscale image is processed according to a preset first pixel threshold to obtain an intermediate binary image; Using the intermediate binary image as a guide image, the grayscale image is subjected to guided filtering to obtain a filtered image; Based on a preset second pixel threshold, high-value pixels in the filtered image are determined, wherein the grayscale value of the high-value pixels is greater than the second pixel threshold; The grayscale values ​​of the high-value pixels are augmented according to a preset augmentation coefficient to obtain an augmented image. The augmented image is then sharpened to obtain a clear image; The contrast of the clear image is adjusted to obtain the binary image.

4. The image processing method according to claim 1, characterized in that, The step of inputting the reliability scores corresponding to all attribution maps into the softmax function and scaling them to obtain the weights corresponding to all prediction results in the prediction result set includes: Formula used: The weights corresponding to all prediction results in the prediction result set are obtained, where w is the weight, T is the temperature factor, and T is adjusted according to the specific application scenario, R... N-1 The reliability score is N, where N is the total number of prediction results in the prediction result set, and N is equal to the number of enhanced images in the first image set. N is greater than or equal to 1, and N-1 is the index of the prediction result.

5. An image processing apparatus for performing the image processing method as described in any one of claims 1-4, characterized in that, include: The image acquisition module is used to acquire the image to be predicted. The binary image acquisition module is used to preprocess the image to be predicted to obtain a binary image; The first image set acquisition module is used to perform data augmentation on the binary image to obtain an enhanced first image set. The prediction result set acquisition module is used to perform data augmentation on the image to be predicted and then input it into a pre-trained deep learning model to obtain the prediction result set. The attribution graph acquisition module is used to perform attribution processing on the prediction result set to obtain the attribution graph set corresponding to the prediction result set. The prediction result acquisition module is used to perform calculations based on the first image set and the attribution map set to obtain the final prediction result.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images